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IN THE UNITED STATES ELECTED/DESIGNATED OFFICE 
OF THE UNITED STATES PATENT AND TRADEMARK OFFICE 
UNDER THE PATENT COOPERATION TREATY-CHAPTER H 
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PRELIMINARY AMENDMENT 

APPLICANTS: Peter Hanselka et al. DOCKET NO: 1 12740-254 

SERIAL NO : GROUP ART UNIT: 

10 EXAMINER: 

INTERNATIONAL APPLICATION NO: PCT/EPOO/003 17 

INTERNATIONAL FILING DATE: 1 7 January 2000 

INVENTION: LOAD DISTRIBUTION METHOD OF A MULTIPROCESSOR 
SYSTEM, AND MULTIPROCESSOR SYSTEM 

15 

Assistant Commissioner for Patents, 
Washington, D.C. 20231 

Sir: 

20 Please amend the above-identified Intemational Application before entry into 

the National stage before the U.S. Patent and Trademark Office under 35 U.S.C. §371 
as follows: 

In the Specification: 

Please replace the Specification of the present application, including the 
25 Abstract, with the following Substitute Specification: 

SPECIFICATION 
TITLE 

LOAD DISTRIBUTION METHOD OF A MULTIPROCESSOR SYSTEM, 
AND MULTIPROCESSOR SYSTEM 
BACKGROUND OF THE INVENTION 
Field of the Invention 

The present invention relates to a method for load distribution in a 
multiprocessor system, in particular in a multiprocessor system of a communication 
system, in which tasks that arise can be processed by a number of processors MP; 



(where i=l,2,...,n) under real-time conditions. The present invention further relates 
to a multiprocessor system, in particular of a communication system, having a load 
distribution mechanism. 
Description of the Prior Art 
5 A similar method for load distribution in a multiprocessor system, in 

particular in a multiprocessor system of a communication system, is disclosed^ for 
example^ in the applicant's European patent application EP 0 645 702 Al. This 
document discloses a method for load balancing in a multiprocessor system, in 
particular a multiprocessor system of a communication system, in which tasks that 
10 arise can be processed by a number of processors under real time conditions, in 
which case, in order to perform the load balancing, the following method steps are 
mentioned: 

- each processor determines its load state in the form of a quantified 
magnitude, 

15 - the load states of the other processors are communicated to each processor 

within a time frame, 

- depending on its load state exceeding a specific magnitude and depending 
on the load states of the remaining processors, each processor outputs at least a 
portion of the tasks arising in it to the remaining processors, and 

20 - the output tasks are divided between the remaining processors in 

accordance with the load states thereof. 

In the example, the method is concretized to the effect that 
distribution quotas are calculated during operation continually and before entry into 
the load distribution^ which, in this case, does not begin tmtil after a specific 

25 overload has been reached, according to which distribution quotas the individual 
processors output their distributable load to other processors in the case of 
overload. If the system is permanently utilized to capacity in a nonuniform manner, 
then the load is distributed only in the event of overload of one or more processors. 
This is accompanied by unnecessary load rejection, however. Reducing the 

30 overload threshold to a lower value does not lead to a satisfactory result because an 
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unnecessarily large amoiint of load is distributed and oscillation states can arise. 
This situation emerges from the assumption made there that the overload or the 
nonuniform loading lasts for a short duration. 

Furthermore, reference is made to Evans D.J. et al., "Dynamic Load 
5 Balancing using Task Transfer Probabilities" in Parallel Computing, Vol. 19, No. 8, 
August 1, 1993, pages 897-916, which also presents a load distribution strategy 
taking account of load distribution probabilities, although exclusively currently 
measured values are taken into consideration. The temporal development of the 
parameters considered is not taken into account. 

10 It is an object of the invention, therefore, to specify an improved load 

distribution method for a multiprocessor system which commences in good time 
and in a "soft" fashion and thereby eliminates permanent unbalanced load states in 
the load offer without load rejection. Moreover, the intention is also to specify a 
corresponding multiprocessor system. 

15 SUMMARY OF THE INVENTION 

Accordingly, the inventors propose a method for load distribution in a 
multiprocessor system, in particular in a multiprocessor system of a communication 
system, in which tasks that arise can be processed by a number of processors MPi 
(where i l,2,...,n) under real-time conditions, having the following iterative 

20 method steps that are repeated at time intervals CI: 

- each processor MPi determines its actual load state Yi and estimates as a 
function of previously communicated distribution quotas qi(old) (where qi = load 
proportion to be distributed, if possible, to other processors MPk) and the typically 
distributable proportion V of a typical task its offered load Aj, which leads to a 

25 multi-value load indication value (balancing indicator) MPbij, 

- each processor MPi indirectly or directly communicates its load indication 
value MPbii to the respective other processors MPk(where k = l,2,...i-l,i+l,...n), 

- each processor MPi determines its load distribution factors pij (where 

j = l,2,...n) as a function of the load indication values MPbik of the other processors 
30 MPk, 
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- each processor MPj determines its distribution quota qi(new) as a function 
of its actual load state Yi and the load distribution factors Pij, 

- on the basis of its quota qj and its load distribution factors pij, each 
processor MPi distributes its distributable load to other processors MPk if its 

5 distribution quota qi(new) exceeds a predetermined value qv. 

Li order to estimate the offered load Ai of a processor MPi, it is 
advantageous to use the formula Ai:=Yi/(l-qiV). A, and Yi can be specified in units 
of erlangs, while the variables qi and V are dimensionless fraction indications in 
accordance with their meaning. 
10 It is also advantageous to subdivide the multi-value load indication value 

(balancing indicator) MPbii into three discrete values, preferably the following 
demarcation with threshold values holding true: NORMAL for MPbii if the 
processor capacity utilization is from 0 to 70%, fflGH for MPbij if the processor 
capacity utilization is from 70% to 85%, and OVERLOAD for MPbii if the 
1 5 processor capacity utihzation is above 85%. 

It is also advantageous if a hysteresis is introduced in the case of a load state 
alteration on account of threshold value overshooting or threshold value 
undershooting in the case of rising or falling processor capacity utilization. 

Moreover, it may be advantageous if the load indication value (balancing 
20 indicator) MPbii is subject to a temporal hysteresis with regard to changes and thus 
experiences a certain inertia. Values of 1 to 2 time intervals CI can be assumed as 
hysteresis limit. 

Further advantageous assumptions in the performance of the method 
according to the present invention are: the typical distributable proportion V of a 

25 typical task shall be the average or maximum proportion, and an average or 

maximum processing time of a task shall be assumed as the typical processing time 
of a task. In this case, the respective average value or maximum value of a 
proportion and respectively of a task also can be continually determined during the 
operating time and, if appropriate, be incorporated as a moving value and be 

30 adopted in updated form into the load distribution method. It is favorable here if the 
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time duration over which the moving values are determined is long relative to the 
control interval CI. 

It is also particularly advantageous if the following holds true for the 
predetermined value qv of the distribution quota qi starting from which the 
5 processor MPi distributes distributable load to other processors MPk: 0.05<qv<0.3, 
preferably 0.1<qv<0.25, preferably qv=0.2. 

Furthermore, the method according to the present invention can be 
configured particularly advantageously if the following criteria are satisfied in the 
calculation of the distribution quota qi: 
10 -Pii: = 0 

- if MPbij corresponds to an average load, preferably MPbij = NORMAL, 

the following holds true: 
Pij(new) = pij(old)+pci/n, for j=l,...,n and i^j 

- if MPbij corresponds to a high load, preferably MPbij=HIGH, the 
1 5 following holds true: 

Pij(new) = pij(old)-pc2/n, for j=l,...,n and i^j 

- if MPbij corresponds to an overload, preferably MPbij=OVERLOAD, the 
following holds true: 

Pij(new) = 0 

20 - in which case the pij (j=i,...,n) is normahzed to 1 with the sum psum of the py 

and 

- as initiahzation value at the beginning of the distribution processes, all pij, 
excluding pii, are identical. 

As advantageous numerical values, 0.1<pci<0.5, preferably 0.2<pci<0.3 and 
25 preferably pci=0.25 may be assumed for the constant pd- Equally, it is 

advantageous to set 0.1<pc2<0.5, preferably 0.2<pc2<0.3, preferably pc2=0.25 for the 
constant pc2. Moreover, the initiahzation value of the pij at the beginning of the 
distribution processes can be set to be equal to (n-1)"^ 

Furthermore, the method according to the present invention can be 
30 configured particularly advantageously if each processor Mpi determines a multi- 
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value load status (load state) MPlSi on the basis of its actual current load Yi, and 
the following criteria are satisfied in the calculation of the load indication values 
MPbii: 

- if MPlsi corresponds to the highest load, preferably MPlSi=EXTREME, the 
5 following holds true: 

qi(new)=Cqi, 

- if psum ^ 1 holds true: 

- if the actual load state Yi is greater than a predetermined value thresholdR, 
qi is increased where qi=min{qi+Cqi,l}, 

10 - if the actual load state Yi is less than a predetermined value thresholdN, qi 

is decreased where qi=max {qi-Cq2, Cqs}, where 0<Cq3<qv, preferably Cq3=0.1, 

- otherwise (thresholdN < Yi < thresholdn), qi obtains an intermediate value 
between the two ahematives mentioned above, preferably by linear interpolation 

- if Psum ^ 1 holds true: qi(new)=qi(old) * psum- 

1 5 With regard to the multi-value load status (load state) MPlsi, the assumption 

of four discrete values is proposed as being particularly preferred, the followmg 
preferably being assumed: NORMAL for MPls, if the processor capacity utilization 
Hes below 70%, HIGH for MPlSi if the processor capacity utilization is from 70% to 
85%, OVERLAND for MPlsi if the processor capacity utilization hes above 85%, 

20 and EXTREME for MPlsi if the load state OVERLAND permanently prevails, hi 
this case, two, is maybe advantageous if the load status (load state) MPlsi is subject 
to a hysteresis with regard to changes. Values of 1 to 2 time intervals CI 
advantageously can be assumed as hysteresis limit. 

For optimal configuration of the method of the present invention, the 

25 following ranges of numbers of numerical values are preferred for the constant Cqi: 
0.05<cqi<0.3, preferably 0.1<Cqi<0.2, preferably Cqi-0.15. Moreover, preferably 
0.05<Cq2<0.2, preferably Cq2=0.10, can be assumed for the constant Cq2. 

With regard to the constant thresholdN, the following is regarded as a 
preferred range of values: 0.6< thresholdN <0.8, preferably thresholdN=0.7. 
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With regard to the constant thresholdn, the following is regarded as a 
preferred range of values: 0.7< thresholdn <0.95, preferably thresholdH=0.85. 

Another configuration of the method according to the present invention 
provides for an overload value OLj of the processors MPi to be additionally 
5 determined in each time interval CI, which value is a measure of the magnitude of 
the overload and serves as a benchmark for overload rejection, where OLi=0,l,...m, 
OLi representing a quantification for the overload of the processor, and the 
distribution quota qi to be increased in any case if OLi>0 where 
qi(new) :=min {qi(old)+Cqi , 1 } . 
10 According to the present invention, it is also possible to adapt the load 

distribution method to changing boundary conditions by the above-specified 
constants (qv, pci, Pc2, qci, qc2, thresholdN, thresholdN, Cqi, Cq2, Cqs) being at least 
partly adapted during operation. 

The present invention additionally proposes a multiprocessor system, in 
1 5 particular of a communication system, having a number of processors MPj (where 
i=l,2,...,n) for executing tasks that arise under real-time conditions, in which case: 

- each processor MPj determines its actual load state Y,, and estimates as a 
function of previously communicated distribution quotas qi(old) (where qi=load 
proportion to be distributed, if possible, to other processors MPk) and the typically 

20 distributable proportion V of a typical task its offered load Ai, which leads to a 
multi-value load indication value (balancing indicator) MPbij, 

- each processor MPi indirectly or directly conununicates its load indication 
value MPbii to the respective other processors MPk(where k = l,2,...i-l,i+l,...n), 

- each processor MPi determines its load distribution probabilities pij (where 
25 j = l,2,...n) as a function of the load indication values MPbik of the other processors 

MPk, 

- each processor MPi determines its distribution quota qi(new) as a function 
of its actual load state Y; and 

- each processor MPi distributes, on the basis of its quota qi and its load 
30 distribution factors pij, 



7 



its distributable load to other processors MPk if its distribution quota qi(new) 
exceeds a predetermined value qy. 

According to the present invention, the multiprocessor system proposed 
above can be configured such that one of the abovementioned methods is 
5 implemented in each case, the implementation being effected by corresponduig 
programming of the processors. 

It should also be pointed out that the index (old) relates, in each case, to the 
values of the preceding iteration step, and the index (new) relates to the now current 
iteration step. 

10 The particular advantage of the method according to the present invention 

and of the corresponding multiprocessor system is that, in contrast to the prior art 
mentioned in the introduction, it ensures a "soft" entry into the load distribution 
and, as a result, is more adaptable and less susceptible to unbalanced load situations 
wherein oscillation states are better avoided. Ultimately, this reduces the probability 
15 of the rejection of tasks, in particular switching tasks. 

Additional features and advantages of the present invention are described in, 
and will be apparent from, the following Detailed Description of the Preferred 
Embodiments and the Drawings. 

DESCRIPTION OF THE DRAWINGS 
20 Figure 1 shows a flow diagram of an arising and distributed load offer; 

Figure 2a shows a graphical illustration of the decisions for updating the 
load distribution factors py; 

Figure 2b shows a graphical illustration of the decisions for updating the 
distribution quotas qi; and 
25 Figure 3 shows a formula for linear interpolation of qi. 

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 
The method according to the present invention (normal load 
balancing=NLB) is a load balancing method in which quotas are set and which runs 
on a multiprocessor system, in particular in a switching center of a communication 
30 system, for distributing operating loads that arise between the respective other 
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processors. It is intended to ensure that lengthy unbalanced load situations are 
eliminated, and as far as possible, all requested tasks are processed in the shortest 
possible time. A particularly advantageous embodiment of this method will be 
described below. 

5 Each processor MPi where i=l ,2,.. .,n carries a distribution quota qi, which 

fixes the proportion V of the distributable load which is actually to be distributed. 
Such a quota enables a softer entry or exit fi-om the load distribution to other 
processors. Oscillation states and load fluctuations are avoided in this way. This 
may be the case, for example, if a processor distributes so much load to another 

1 0 processor that the latter is in turn overloaded. 

The distribution quota qi is determined anew at each time interval CI. The 
only information required by the other processors MPk where k=l,...i-l,i+l,...n for 
each CI are load value indicators (balancing indicators) MPbii. These load value 
indicators are, similarly to the load status values (load states)fi-om the load control, 

1 5 load states having the significances NORMAL, HIGH or OVERLOAD. While the 
load state is determined on the basis of the actually processed load Yj of the 
processor MPi, the load value indicator MPbii is determined firom an estimation of 
the currently offered load Ai. The estimated offered load Ai may, due to load 
distribution, be considerably more than the actually processed load Yi and 

20 constitutes the crucial quantity which (in the form of the load value indicator 

MPbii) is made available as information by one processor MPj to the others MPk- 

In addition to the distribution quota qi, each MPi carries probabilities pij 
which indicate the probability that, in the event of load distribution, load will be 
transferred from the i-th processor MPi to the j-th processor MPj. The probabilities 

25 are determined in such a way that if, for instance, the j-th processor MPj already has 
a large amount of load to process and, therefore, can only take up a small amount of 
additional load, the associated pij is less than the Pik for a firee MPk. 

Figure 1 illustrates the interaction of the pij and qj. The double indexing "ij" 
of the characteristic quantities refers to the respective processor with the number of 

30 the first index (here i) in each case knowing a "column" of n values with the second 
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index (here j). It should be noted that each processor only knows its relevant values 
(that is to say its column), overall a square matrix being known in the system. Thus, 
for example, pij is the probability that load will be distributed from the i-th MP to 
the j-th MP if the i-th MP has too much load. 
5 In Figure 1 , moreover, the actually processed load of the j -th processor MPj 

is designated by Yj, the estimated offered load is designated by Aj and that part of 
the load offer which can be shifted is designated by a. The load situation shown is 
overload (OVERLOAD) on MPi, there still being space for additional tasks on the 
MPk where k=2,3,4. Figure 1 shows how the MPi processes a first part of the load 
1 0 itself and distributes the remainder a. Of this remainder a, the largest proportion 
goes to MPs and the smallest proportion to MP4, which, in this example, already 
has a large amount of its own load to process. The loads which the MPk additionally 
receive besides that from MPi are not depicted. The width of the flow bars 
represents a measure of the magnitude of the load. 
1 5 The following algorithm is thus produced in accordance with the concept of 

the present invention: if the j-th processor MPj reports the balancing indicator 
NORMAL, the pij is increased on the MPi respectively considered. The probability 
that this processor MPi will output load to MPj if it has to distribute load thus rises. 
If the balancing indicator HIGH is reported, then the pij is decreased. If the 
20 balancing indicator OVERLOAD is reported, py is set to zero, with the result that 
no load is output to the j-th processor MPj. The distribution quota qi is changed 
following the determination of the pij. If many of the pij were able to be increased, 
then the sum of the pij over j is greater than 1 and there is evidently still space on 
the other processors MPk- The distribution quota qi can^ thus^ be changed according 
25 to the requirements of the processor (considered) . 

The distribution quota qi is increased in the event of high load Yj on the processor 
MPi considered, and qi is decreased in the event of low load. If many of the pij have 
been reduced, then the sum of the pij over j is less than 1 and the distribution quota 
qi must be reduced. 
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An illustration of these decisions is represented in Figures 2 and 2b. The 
decision diagrams show the updating algorithms for py (Figure 2a) and for 
distribution quota qi (Figure 2b), which are carried out in each time interval CI for 
the i-th processor MPi. 

In the load distribution method (NLB) according to the present invention, 
some parameters (constants) are required, the choice of which can greatly influence 
the behavior in specific load situations. In most cases this results in a conflict 
between a load distribution method which can react rapidly to load changes, and a 
stable load distribution method which does not tend toward oscillations and further 
distribution of tasks. In this case, "further distribution" refers to the simultaneous 
distribution of dedicated load and the processing of extrinsic load on one processor. 

The following parameter changes bring about a more rapidly reacting NLB: 

- the relatively great alteration of qi where: 0.15<Cqi, O.Kcqz 

- the relatively great alteration of the pij where: 0.25<pci, 0.25<pc2 

- the relatively late setting of the load indication values MPbij where: 
thresholdH>0.7 (i.e. report only in the event of relatively high load 'HIGH') to the 
other processors MPk. 

m detail, the preferred method thus proceeds as follows in a multiprocessor 
communication computer: , 

As the duration of the time interval (control interval) CI of the time fi-ame 
with which the method iteratively proceeds, 1 to 2 seconds is preferably chosen in 
the case of the presently known multiprocessor systems appertaining to switching 
technology. It goes without saying that the time interval can be shortened with 
rising processor power. 

The quantities qi, pij, MPlSi and MPby are updated in each control interval 

CI. 

The actually processed load Yi of a processor MPi is determined as 
processor run time quantity, measured in erlangs. 
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The estimated offered load Aj of a processor MPi is determined from the 
distribution quota qi of the current control interval CI and the estimated 
distributable proportion of an average task, for example the processing of a call. 

The following holds true: 
5 The number of processors MPj in the multiprocessor system is n. 

Ai := Yi/(l-qiV), where V is the distributable proportion of a call. 

MPlsi: load state of the i-th MP, can assume the values NORMAL, HIGH, 
OVERLOAD or EXTREME. The actually processed load Yi is used to calculate 
the load state. 

1 0 In order to avoid premature changes of the MPlSj, hystereses are introduced. 

If, for instance, the MPlsi is set from NORMAL to HIGH, it must be the case that 
Yi > thresholdN + A+, whereas, in order to get from HIGH to NORMAL, it must be 
the case that Yi < thresholdN - A.. This procedure is also known as the high water- 
low water method. In the case of EXTREME, the distribution method (load 

1 5 balancing) must be switched off for this processor MPi, for system engineering 
reasons relating to the switching center. 

thresholdN: is the normal load threshold - after taking a hysteresis into 
account, the MPls is recorded as NORMAL below the said threshold and as HIGH 
above said threshold. 

20 thresholdH: High load threshold - after taking a hysteresis and a load- 

dependent temporal delay (start indicator) into account, the MPls is recorded as 
HIGH below this threshold and as OVERLOAD above said threshold. 

The load indication value (balancing indicator) MPbii of the i-th processor 
MPi can assume the values NORMAL, HIGH or OVERLOAD. This value is 

25 calculated like the MPlsi, except that here, instead of the actual load Yi, the 

estimated offered load Ai is taken as a basis and other values are adopted for A+ and 
■ A., where A+ = A. = 0.02. 

In addition, an Overload Level OLi of the processor MPi is determined, 
which can assume the values 0... 6 and is conceived as quantification of the 
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overload state of the processor MPi. If the OLi>0, calls are rejected; the higher the 
value, the greater the probability that a call will be rejected. 

The load which is to be distributed from MPi to MPj is expressed as a 
probability pij and can, thus, assume values between 0 and 1. 

The magnitude of the value py is determined by the following criteria: 

- initialize pij where pij:=(n-l)'^ 

- piii 0, MPi should not distribute to itself. 

- If MPbij=NORMAL: pij Pij + 0.25/n, j=l,...,n, i^^j. The old pij can be 
increased because there is still space on the processor MPj. 

- If MPbij=HIGH: pij pij - 0.25/n. The old pij must be decreased because 
MPj is utilized to full capacity. 

- If MPbij=OVERLOAD: pij = 0. No load should be output to overloaded 

processors MPn- 

The newly determined pij must still be normaUzed: 

Set psum = sum (pij) over j=l ,...,n and normalize (if psum > 0) where 

Pij ~^ Pij/Psum 

Afterward, the distribution quota qi is determined using the following 
criteria: 

- Initialization value: qi = 0.1 

- If the MPlsi ^EXTREME: qi = 0.1. This MP is overloaded so severely that 
even its own proportion for a distributed call would overtax it. Therefore, no load 
balancing, rather only rejecting; load balancing is not practical, moreover, for 
system engineering reasons'relating to the switching center. 

- If Psum > 1, more load can evidently be distributed, qi can then be 
determined according to the requirements of the MPi, where: 

1. If the OLi > 0, increase qi in any case, where: qi -> min {qi + 0.15, 1} 

2. If Yi > thresholdn, increase qi, where: qi min {qi + 0.15, 1} 

3. If Yi < thresholdN, decrease qi, where: qi max {qi - 0.10, 0.1} 

4. Otherwise, if thresholdN < Yi < thresholdn the following holds true: 
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qi -> min{max{qi+(0.25/thresholdH-thresholdN)) * (Yi-thresholdN)-0.1, 0.1), 

1.0}. 

This is the linear interpolation between the above increase by 0.15 and the 
above decrease by 0.1. The formula is represented again more readably in Figure 3. 

- If psum< 1, evidently too much load was distributed and qi must be 
decreased, where: qi qi * psum- 

- The processor MPi distributes load to other processors MPk if it becomes 
the case that qi> 0.25. 

The method according to the present mvention thus has the following 
properties and advantages: 

A very small information overhead exists between the processors 
participating in the load distribution method. Only a few, preferably three-value, 
load states are reciprocally known, which load states are updated and distributed 
only once per control interval. 

For each processor, there is a quota which is updated in each control interval 
and regulates the proportion of the load which is to be distributed from the 
processor considered to the other processors involved. 

For each processor, there are individual regulators which divide between the 
other processors the load that is to be distributed. 

The method is designed as a "fire-fighting measure" which only takes effect 
when a processor reaches overload and, if appropriate, tasks (calls) are rejected, 
rather the load distribution commences earher and in a softer fashion. As a result, 
continuous unbalanced load states can be processed better and with fewer rejected 
tasks (calls). 

In the method according to the present invention, the load states which are 
distributed to the other processors are consistently determmed on the basis of the 
estimated offered load and not on the basis of the actually processed load. 

The present method does not require a load balancing flag which regulates 
entry into the load distribution. The entry is regulated by way of the distribution 
quota qi. Furthermore, mutual dependencies between the load states and the load 
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balancing flag have been eliminated as a result of the absence of a load balancing 
flag. As a result, the algorithm more easily can be subsequently adapted to changed 
conditions. 

The load-dependent alteration of the individual regulators (load distribution 
5 factors pij) takes place as a function of the nimiber n of processors participating in 
the load distribution. Consequently, the method is independent of the number of 
processors involved. 

The load-dependent alteration of the distribution quotas and of the 
individual regulators per control interval takes place in such a way as to avoid 
10 excessively slow "creeping" to the optimum value. 

The load-dependent alteration of the individual regulators prevents the 
values from staying at the setting of the preceding load distribution period during a 
period without load distribution. Rather, there is regulation back to an initial 
setting. 

15 The inertia, known from the prior art, in the alteration of the quotas has 

been removed in order to enable easier tracking to the load situation that is actually 
present. 

Attention is supplementary drawn to the definition of a few terms in this 
application: 

20 The term or word element "quota" describes the fraction of a whole with a range of 
values between 0 and 1 . 

The term or word element "state" describes the instantaneous situation or 
the instantaneous ciirrent value of a quantity. Thus, e.g., the load state of a 
processor is to be understood as the value of the current load of the processor. 

25 The term hysteresis defines the dependence of a current quantity on its 

earlier values, its history or its time characteristic. 

Although the present invention has been described with reference to specific 
embodiments, those of skill in the art will recognize that changes may be made 
thereto without departing from the spirit and scope of the invention as set forth in 

30 the hereafter appended claims. 
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ABSTRACT OF THE DISCLOSURE 

A method for load distribution in a real-time multiprocessor system and to a 
multiprocessor system, each processor carrying a distribution quota which fixes the 
proportion of the distributable load which is actually to be distributed. The 
5 distribution quota is determined anew at time intervals. The only information 
required by the other processors for each time interval are load value indicators, 
which depend on an estimated load. Probabilities indicating how load is transferred 
from one processor to the others during load distribution are additionally carried. 
Afterward, on the basis of its distribution quota and its load distribution factors, 
10 each processor distributes its distributable load to other processors if it distribution 
quota exceeds a predetermined value. 
In the claims: 

On page 19, cancel line 1, and substitute the following left-hand justified 
headmg therefor: 
15 We Claim as Our Invention; 

Please cancel claims 1-26, without prejudice, and substitute the following 
claims therefor: 

27. A method for load distribution in a multiprocessor system of a 
communication system, in which tasks that arise can be processed by a plurality of 
20 processors MPi (where i = 1 ,2,...,n) under real-time conditions, the method 
comprising the following steps that are repeated at time intervals CI: 

determining, via each processor MPi, a respective actual current load Yi; 

estimating, via each processor MPi and as a function of previously 
communicated distribution quotas qi(old) and a typically distributable proportion V 
25 of a typical task, a respective offered load Ai, which leads to a multi-value load 
indication value MPbii, the distribution quota qi representing a load proportion 
which can be distributed to other processors MPk; 

communicating, via each processor MPi, the respective load indication 
value MPbii to the respective other processors MPk (where k = l,2,...i-l,i+l,...n); 
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determining, via each processor MPi, respective load distribution 
probabilities pij (where j = 1,2,.. .n) as a ftmction of the load indication values MPbik 
of the other processors MPt; 

determining, via each processor MPj, a new distribution quota qi(new) as a 
5 function of the respective current load Yi and the load distribution factors Py-; and 

distributing, via each processing MPj and based on the respective new 
distribution quota qi and the respective load distribution factors py, the respective 
distributable load to the other processors MPk if the respective new distribution 
quota qi(new) exceeds a predetermined value qv. 

10 

28. A method for load distribution in a multiprocessor system of a 
conmimication system as claimed in claim 27, wherein the estimated offered load 
Ai of the processor MPj is calculated according to the formula Ai:=Yi/(l-qiV). 

15 29. A method for load distribution in a multiprocessor system of a 

communication system as claimed in claim 27, wherein the multi-value load 
indication value MPbi; can assume three discrete values, NORMAL, HIGH and 
OVERLOAD, where NORMAL corresponds to a processor capacity utilization of 
from 0 to 70%, HIGH corresponds to a processor capacity utilization of from 70% 

20 to 85% and OVERLOAD corresponds to a processor capacity utilization of from 
85% to 100%. 

30. A method for load distribution in a multiprocessor system of a 
communication system as claimed in claim 29, wherein the multi-value load 

25 indication value MPbii is subject to a hysteresis with regard to changes. 

31. A method for load distribution in a multiprocessor system of a 
commimication system as claimed in claim 27, wherein one of an average and a 
maximum distributable proportion of a typical task CallP is regarded as the typical 

30 distributable proportion V. 
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32. A method for load distribution in a multiprocessor system of a 
communication system as claimed in claim 31, wherein the one of the average and 
the maximum distributable proportion of a typical task is continually determined as 
moving average and moving maximum value, respectively, over a predetermined 

5 time period Id. 

33. A method for load distribution in a multiprocessor system of a 
communication system as claimed in claim 32, wherein the following holds true for 
the predetermined time period to: to » CI. 

10 

34. A method for load distribution in a multiprocessor system of a 
communication system as claimed in claim 31, wherein one of an average and a 
maximum task is assumed as the typical task. 

15 35. A method for load distribution in a multiprocessor system of a 

communication system as claimed in claim 34, wherein the one of the average and 
the maximum task is continually determined as moving average and moving 
maximum value, respectively, over a predetermined time period tp. 

20 36. A method for load distribution in a multiprocessor system of a 

communication system as claimed in claim 35, wherein the following holds true for 
the predetermined time period to: to » CI. 

37. A method for load distribution in a multiprocessor system of a 

25 communication system as claimed in claim 27, wherein the following holds true for 
the predetermined value qv of the distribution quota qi starting from which the 
processor MPi distributes distributable load to other processors MPk: 0.05<qv<0.3. 

38. A method for load distribution in a multiprocessor system of a 

30 commimication system as claimed in claim 27, wherein the following holds true for 
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the predetermined value qy of the distribution quota qi starting from which the 
processor MPi distributes distributable load to other processors MPk: 0.1<qv<0.25. 

39. A method for load distribution in a multiprocessor system of a 

5 communication system as claimed in claim 27, wherein the following holds true for 
the predetermined value qy of the distribution quota qj starting from which the 
processor MPi distributes distributable load to other processors MPk: qv= 0.2. 

40. A method for load distribution in a multiprocessor system of a 
communication system as claimed in claim 29, wherein the calculation of the 
distribution quota qi satisfies the following criteria: 

Pii: = 0 

if MPbij corresponds to an average load, preferably MPbij = NORMAL, the 

following holds true: 
pij(new) = pij(old)+pci/n, for j=l,..., and i^g; 

if MPbij corresponds to a high load, preferably MPbij=HIGH, the following 
holds true: 

Pij(new) = pij(old)-pc2/n, for j=l,...,n and ij^y, 

if MPbij corresponds to an overload, preferably MPbij=OVERLO AD, the 
following holds true: 
Pij(new) = 0; 

in which case the py (j=i,...,n) are normalized to 1 with the sum psum 

of thepij; 

and 

as initiahzation value at the beginning of the distribution processes, all py, 
excluding pii, are identical. 

41 . A method for load distribution in a multiprocessor system of a 
communication system as claimed in claim 40, wherein the following holds true for 
the constant pci : 

30 0.1<pci<0.5. 
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42. A method for load distribution in a multiprocessor system of a 
communication system as claimed in claim 40, wherein the following holds true for 
the constant pcu 

5 0.2<pci<0.3. 

43. A method for load distribution in a multiprocessor system of a 
commimication system as claimed in claim 40, wherein the following holds true for 
the constant pci; 

10 pel = 0.25. 

44. A method for load distribution in a multiprocessor system of a 
communication system as claimed in claim 40, wherein the following holds true for 
the constant pci-" 0.1<pc2<0.5 

15 

45 . A method for load distribution in a multiprocessor system of a 
communication system as claimed in claim 40, wherein the following holds true for 
the constant pc2: 0.2<pc2<0.3. 

20 46. A method for load distribution in a multiprocessor system of a 

communication system as claimed in claim 40, wherein the following holds true for 
the constant pc2 = 0.25. 

47. A method for load distribution in a multiprocessor system of a 

25 communication system as claimed in claim 40, wherein the initialization value of 
the pij at the beginning of the distribution processes is set to be equal to (n-1)'^ 

48. A method for load distribution in a multiprocessor system of a 
communication system as claimed in claim 40, wherein each processor MPj 

30 determines a multi-value load status MPls, based on the respective actual current 
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load Yi, and the calculation of the load indication values MPbii satisfies the 
following criteria: 

if MPlsj corresponds to the highest load, the following holds true: 
qi(new)=Cqi; 
5 if psum ^ 1 holds true: 

- if the actual current load Yj is greater than a predetermined 
value thresholdn, qi is increased where qi=min{qi+Cqi,l}, 

- if the actual current load Yi is less than a predetermined value 
thresholdN, qi is decreased where qi=max{qi-Cq2, Cqs}, where 0<Cq3<qv, preferably 

10 Cq3=0.1, 

-if thresholdN ^ Y; < thresholdn, qi obtains an intermediate 
value between the two alternatives mentioned above by linear interpolation; and 
if Psum ^ 1 holds true: qi(new)=qi(old) * psum- 

15 49. A method for load distribution in a multiprocessor system of a 

communication system as claimed in claim 48, wherein the multi-value load status 
MPlSi is subject to a hysteresis with regard to changes. 

50. A method for load distribution in a multiprocessor system of a 
20 communication system as claimed in claim 48, wherein the multi-value load status 
MPIsi can assume four discrete values, NORMAL (0 to 0.7), HIGH (0.7 to 0.85), 
OVERLOAD (0.85 to 1) and EXTREME (if load status over a plurality of CI 
OVERLOAD). 

25 5 1 . A method for load distribution in a multiprocessor system of a 

communication system as claimed in claim 49, wherein the following holds true for 
the constant Cqi: 0.05<Cqi<0.3. 
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52. A method for load distribution in a multiprocessor system of a 
communication system as claimed in claim 49, wherein the following holds true for 
the constant Cqi<0.2. 

5 53. A method for load distribution in a multiprocessor system of a 

communication system as claimed in claim 49, wherein the following holds true for 
the constant Cqi=0.15, 

54. A method for load distribution in a multiprocessor system of a 
communication system as claimed in claim 49, wherein the following holds true for 
the constant Cqi' 0.05<Cq2<0.2. 

55. A method for load distribution in a multiprocessor system of a 
communication system as claimed in claim 49, wherein the following holds true for 
the constant Cq2=0.10. 

56. A method for load distribution in a multiprocessor system of a 
communication system as claimed in claim 49, wherein the following holds true for 
the constant thresholdN: 0.6 < thresholdN < 0.8. 

57. A method for load distribution in a multiprocessor system of a 
communication system as claimed in claim 49, wherein the following holds true for 
the constant thresholdN = 0.7. 

25 58. A method for load distribution in a multiprocessor system of a 

communication system as claimed in claim 49, wherein the following holds true for 
the constant thresholdn: 0.7 < thresholdn < 0.95. 
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59. A method for load distribution in a multiprocessor system of a 
communication system as claimed in claim 49, wherein the following holds true for 
the constant thresholdn = 0.85. 



5 60. A method for load distribution in a multiprocessor system of a 

communication system as claimed in claim 27, wherein an overload value OLi of 
the processors MPi is additionally determined for the purpose of quantifying the 
overload state of the processors, which overload value is a measure of the 
magnitude of the overload, where OLi=0,l,...m and the distribution quota qi is 
1 0 increased if the magnitude of OLj becomes greater than 0 and 
qi(new):=min{qi(old)+Cqi,l} is set. 

61 . A multiprocessor system of a communication system, having a 
plurality of processors MPj (where i=l,2,...,n) for executing tasks that arise under 
1 5 real-time conditions, each of the plurality of processors comprising: 

means for determining a respective actual current load Yf and for estimating 
as a function of previously communicated distribution quotas q^old) and a typically 
distributable proportion V of a typical task a respective offered load Aj, which leads 
to a multi- value load indication value MPbij, the distribution quota qi representing a 
20 load proportion which can be distributed to other processors MPk; 

means for communicating the respective multi-value load indication value 
MPbii to the other processors MPk (where k = l,2,...i-l,i+l,...n); 

means for determining respective load distribution probabilities py (where 
j = l,2,...n) as a function of the load indication values MPbik of the other processors 
25 MPk; 

means for determining a new distribution quota qi(new) as a function of the 
respective actual current load Yi; and 

means for distributing, on the basis of the quota qi and the load distribution 
factors pij, a respective distributable load to the other processors MPk if the new 
30 distribution quota qi(new) exceeds a predetermined value qy. 
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REMARKS 



The present amendment makes editorial changes and corrects typographical 
errors in the specification, which includes the Abstract, in order to conform the 
specification to the requirements of United States Patent Practice. No new matter is 
added thereby. Attached hereto is a marked-up version of the changes made to the 
specification by the present amendment. The attached page is captioned " Version 
With Markings To Show Changes Made". 

In addition, the present amendment cancels original claims 1-26 in favor of 
new claims 27-61 . Claims 27-61 have been presented solely because the revisions 
by red-lining and underlining which would have been necessary in claims 1-26 in 
order to present those claims in accordance with preferred United States Patent 
Practice would have been too extensive, and thus would have been too burdensome. 
The present amendment is intended for clarification purposes only and not for 
substantial reasons related to patentability piirsuant to 35 USC §§103, 102, 103 or 
112. Indeed, the cancellation of claims 1-26 does not constitute an intent on the 
part of the Applicants to surrender any of the subject matter of claims 1-26. 

Early consideration on the merits is respectfiiUy requested. 



Wilham E. Vanghan 
Bell, Boyd (SlLloyd LLC 
P.O. Box 1135 

Chicago, Illinois 60690-1135 
(312) 807-4292 
Attorneys for Applicants 




RespectfiiUy submitted, 



(Reg. No. 39,056) 
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JG17R8C'dPCT/PrO 2 3juLaoa^ 

VERSIONS WITH MARKINGS TO SHOW CHANGES MADE 
In The Specification: 

The Specification of the present application, including the Abstract, has been 
amended as follows: 

SPECIFICATION 
TITLE 

5 Load distribution method of a multiprocessor syst e m, and multiprocessor 

LOAD DISTRIBUTION METHOD OF A MULTIPROCESSOR SYSTEM, 
AND MULTIPROCESSOR SYSTEM 
BACKGROUND OF THE INVENTION 

10 Description 

Field of the Invention 

The present invention relates to a method for load distribution in a 
multiprocessor system, in particular in a multiprocessor system of a communication 
system, in which tasks that arise can be processed by a plurality number of 

1 5 processors MPi (where i= 1 ,2, . . . ,n) under real-time conditionsraBd . The present 
invention further relates to a multiprocessor system, in particular of a 
communication system, having a load distribution mechanism. 
Description of the Prior Art 

A similar method for load distribution in a multiprocessor system, in 

20 particular in a multiprocessor system of a communication system, is disclosed^ for 
example^ in the applicant's European patent application EP 0 645 702 Al . This 
document discloses a method for load balancing in a multiprocessor system, in 
particular a multiprocessor system of a communication system, in which tasks that 
arise can be processed by a plu r al it y number of processors under real time 

25 conditions, in which case, in order to perform the load balancing, generally the 
following method steps are mentioned: 

- each processor determines its load state in the form of a quantified 
magnitude. 
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- the load states of the other processors are comniTinicated to each processor 
within a time frame, 

- depending on its load state exceeding a specific magnitude and depending 
on the load states of the remaining processors, each processor outputs at least a 

5 portion of the tasks arising in it to the remaining processors, and 

- the output tasks are divided between the remaining processors in 
accordance with the load states thereof. 

hi the exemplary embodiment example , the method is concretized to 
the effect that distribution quotas are calculated during operation continually and 

10 before entry into the load distribution^ which^ in this case^ does not begin imtil after 
a specific overload has been reached, according to which distribution quotas the 
individual processors output their distributable load to other processors in the case 
of overload. If the system is permanently utilized to capacity in a nonuniform 
manner, then the load is distributed only in the event of overload of one or more 

1 5 processors. This is accompanied by unnecessary load rejection, however. Reducing 
the overload threshold to a lower value does not lead to a satisfactory result because 
fees an unnecessarily large amount of load is distributed and oscillation states can 
arise. This situation emerges from the assumption made there that the overload or 
the nonuniform loading lasts for a short duration. 

20 Furthermore, reference is made to Evans D.J. et al., "Dynamic Load 

Balancing using Task Transfer Probabilities" in Parallel Computing, Vol. 19, No. 8, 
August 1, 1993, pages 897-916, which also presents a load distribution strategy 
taking account of load distribution probabilities, although exclusively currently 
measured values are taken into consideration. The temporal development of the 

25 parameters considered is not taken into account. 

It is an object of the invention, therefore, to specify an improved load 
distribution method for a multiprocessor system which commences in good time 
and in a "soft" fashion and thereby eliminates permanent imbalanced load states in 
the load offer without load rejection. Moreover, the intention is also to specify a 

30 corresponding multiprocessor system. 
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SUMMARY OF THE INVENTION 



The object is achieved on the one hand by means of a method having the 
method steps of the first method claim and on the other hand by means of a 
multiprocessor system having the features of the first apparatus claim. 

Accordingly, the inventors propose a method for load distribution in a 
multiprocessor system, in particular in a multiprocessor system of a communication 
system, in which tasks that arise can be processed by a plurality number of 
processors MPi (where i = 1,2,. ..,n) under real-time conditions, having the 
following iterative method steps that are repeated at time intervals CI: 

- each processor MP,- determines its actual load state Yj and estimates as a 
fimction of previously communicated distribution quotas qi(old) (where qi = load 
proportion to be distributed, if possible, to other processors MPk) and the typically 
distributable proportion V of a typical task its offered load A;, which leads to a 
multi-value load indication value (balancing indicator) MPbii, 

- each processor MPj indirectly or directly commvmicates its load indication 
value MPbii to the respective other processors MPk (where k = l,2,...i-l,i+l,...n), 

- each processor MPi determines its load distribution factors pij (where 
j = 1,2,. ..n) as a fimction of the load indication values MPbik of said &e other 
processors MPk, 

- each processor MPj determines its distribution quota qi(new) as a fimction 
of its actual load state Yj and the load distribution factors Py, 

- on the basis of its quota qi and its load distribution factors pij, each 
processor MPi distributes its distributable load to other processors MPk if its 
distribution quota qi(new) exceeds a predetermined value qy. 

In order to estimate the offered load Aj of a processor MPi, it is 
advantageous to use the formula Ai:=Yi/(l-qiV). Ai and Yi can be specified in units 
of erlangs, while the variables qi and V are dimensionless fi-action indications in 
accordance with their meaning. 

It is also advantageous to subdivide the multi- value load indication value 
(balancing indicator) MPbij into three discrete values, preferably the following 
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demarcation with threshold values holding true: NORMAL for MPbii if the 
processor capacity utilization is from 0 to 70%, HIGH for MPbij if the processor 
capacity utilization is from 70% to 85%, and OVERLOAD for MPbi; if the 
processor capacity utilization is above 85%. 
5 It is also advantageous if a hysteresis is introduced in the case of a load state 

alteration on accovmt of threshold value overshooting or threshold value 
undershooting in the case of rising or falling processor capacity utilization. 

Moreover, it may be advantageous if the load indication value (balancing 
indicator) MPbii is subject to a temporal hysteresis with regard to changes and thus 
10 experiences a certain inertia. Values of 1 to 2 time intervals CI can advantageously 
be assumed as hysteresis limit. 

Further advantageous assumptions in the performance of the method 
according to the present invention are: the typical distributable proportion V of a 
typical task shall be the average or maximum proportion, and an average or 
15 maximum processing time of a task shall be assumed as the typical processing time 
of a task. In this case, the respective average value or maximum value of a 
proportion and respectively of a task eaa also can advantageously be continually 
determined during the operating time and, if appropriate, be incorporated as a 
moving value and be adopted in updated form into the load distribution method. It 
20 is favorable here if the time duration over which the moving values are determined 
is long relative to the control interval CI. 

It is also particularly advantageous if the following holds true for the 
predetermined value qv of the distribution quota qi starting from which the 
processor MPi distributes distributable load to other processors MPk: 0.05<qv<0.3, 
25 preferably 0. 1 <qv<0.25, preferably qv=0.2. 

Furthermore, the method according to the present invention can be 
configured particularly advantageously if the following criteria are satisfied in the 
calculation of the distribution quota qj: 

-Pii: = 0 
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- if MPbij corresponds to an average load, preferably MPbij = NORMAL, 

the following holds true: 
pij(new) = pij(old)+pci/n, for j==l,...,n and i^j 

- if MPbij corresponds to a high load, preferably MPbij=HIGH, the 
5 following holds true: 

pij(new) = pij(old)-pc2/n, for j=l,...,ii and i^q 

- if MPbij corresponds to an overload, preferably MPbij^OVERLOAD, the 
following holds true: 

pij(new) = 0 

10 - in which case the pij (j=i,...,n) is normalized to 1 with the sum psum of the py 

and 

- as initialization value at the beginning of the distribution processes, all pij, 
excluding pii, are identical. 

As advantageous numerical values, 0.1<pci<0.5, preferably 0.2<pci<0.3 and 
15 preferably pci=0.25 may be assumed for the constant pd. Equally, it is 

advantageous to set 0.1<pc2<0.5, preferably 0.2<pc2<0.3, preferably pc2=0,25 for the 
constant pd- Moreover, the initialization value of the py at the beginning of the 
distribution processes can be set to be equal to (n-l)"\ 

Furthermore, the method according to the present invention can be 
20 configured particularly advantageously if each processor Mpi determines a multi- 
value load status (load state) MPlsi on the basis of its actual current load Yi, and 
the following criteria are satisfied in the calculation of the load indication values 
MPbii: 

- if MPlSi corresponds to the highest load, preferably MPlSi=EXTREME, the 
25 following holds true: 

qi(new)=Cqi, 

- if psum ^ 1 holds true: 

- if the actual load state Yj is greater than a predetermined value thresholdn, 
qi is increased where qi=min{qi+Cqi,l}, 
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- if the actual load state Yi is less than a predetermined value thresholdN, qi 
is decreased where qi=max{qi-Cq2, Cqs}, where 0<Cq3<qv, preferably Cq3=0.1, 

- otherwise (thresholdN ^ Yi < thresholdn), qi obtains an intermediate value 
between the two alternatives mentioned above, preferably by linear interpolation 

5 - if psum ^ 1 holds true: qi(new)=qi(old) * psum- 

With regard to the multi-value load status (load state) MPlsi, the assumption 
of four discrete values is proposed as being particularly preferred, the following 
preferably being assumed: NORMAL for MPls, if the processor capacity utilization 
hes below 70%, HIGH for MPlsi if the processor capacity utilization is from 70% to 

10 85%, OVERLAND for MPlsj if the processor capacity utilization lies above 85%, 
and EXTREME for MPlSj if the load state OVERLAND permanently prevails. In 
this case, two, is maybe advantageous if the load status (load state) MPlsi is subject 
to a hysteresis with regard to changes. Values of 1 to 2 time intervals CI eaa 
advantageously cm be assumed as hysteresis limit. 

1 5 For optimal configuration of the method of the present invention , the 

following ranges of numbers of numerical values are preferred for the constant Cqi: 
0.05<Cqi<0.3, preferably 0.1<Cqi<0.2, preferably Cqi=0.15. Moreover, preferably 
0.05<Cq2<0.2, preferably Cq2=0.10^ can be assiuned for the constant Cq2. 

With regard to the constant thresholdN, the following is regarded as a 

20 preferred range of values: 0.6< thresholdN <0.8, preferably thresholdN=0.7. 

With regard to the constant thresholdH, the following is regarded as a 
preferred range of values: 0.7< thresholdn <0.95, preferably thresholdH=0.85. 

Another configuration of the method according to the present invention 
provides for an overload value OL, of the processors MPi to be additionally 

25 determined in each time interval CI, which value is a measure of the magnitude of 
the overload and serves as a benchmark for overload rejection, where OLi=0,l,...m, 
OLi representing a quantification for the overload of the processor, and the 
distribution quota qi to be increased in any case if OLi>0 where 
qi(new):=min{qi(old)+Cqi,l } . 
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According to the present invention, it is also possible to adapt the load 
distribution method to changing boundary conditions by the above- specified 
constants (qv, pci, Pci, qci, <lc2, thresholdN, thresholdN, Cqi, Cq2, Cqs) being at least 
partly adapted during operation. 
5 The present invention additionally proposes a multiprocessor system, in 

particular of a communication system, having a plurality number of processors MP,- 
(where i=l,2,...,n) for executing tasks that arise under real-time conditions, in 
which case: 

- each processor MPi has means for determining determines its actual load 
10 state Yi, and for estimating estimates as a function of previously commimicated 

distribution quotas qi(old) (where qi=load proportion to be distributed, if possible, 
to other processors MPk) and the typically distributable proportion V of a typical 
task its offered load Aj, which leads to a multi-value load indication value 
(balancing indicator) MPbi;, 
15 - each processor MPi has means for indirectly or directly communicating 

communicates its load indication value MPbii to the respective other processors 
MPk (where k = l,2,...i-l,i+l,...n), 

- each processor MPj has means for determining determines its load 
distribution probabilities pij (where j = 1,2,.. .n) as a function of the load indication 

20 values MPbik of said die other processors MPk, 

- each processor MP] has means for determining determines its distribution 
quota qi(new) as a function of its actual load state Yi and 

- each processor MPi has means for distributing distributes , on the basis of 
its quota qi and its load distribution factors pij, 

25 its distributable load to other processors MPk if its distribution quota qi(new) 
exceeds a predetermined value qv. 

According to the present invention, the multiprocessor system proposed 
above can be configured such that one of the abovementioned methods is 
implemented in each case, the implementation being effected by corresponding 

30 programming of the processors. 
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It should also be pointed out that the index (old) relates^ in each case^ to the 
values of the preceding iteration step, and the index (new) relates to the now current 
iteration step. 

The particular advantage of the method according to the present invention 
5 and of the corresponding multiprocessor system is that, in contrast to the prior art 
mentioned in the introduction, it ensures a "soft" entry into the load distribution 
and, as a result, is more adaptable and less susceptible to unbalanced load 
situations7«id wherein oscillation states are better avoided better . Ultimately, this 
reduces the probability of the rejection of tasks, in particular switching tasks. 
10 Further configuration s , additional features and advantages of the invention 

emerge from the following description of a preferred exemplary embodiment with 
reference to the drawings- 
It is xmderstood that the features of the invention that hav e been mentioned 
above and will be explained below can be used not only in the combination 
15 respectively specified but also in other combinations or by themselves, without 
departing from the scope of the invention. 

Additional features and advantages of the present invention are described in, 
and will be apparent from, the following Detailed Description of the Preferred 
Embodiments and the Drawings. 
20 DESCRIPTION OF THE D RAWINGS 

Specifically, in the figures: 

Figure U shows a flow diagram of the m arising and distributed load offer; 
Figure 2aT shows a graphical illustration of the decisions for updating the 
load distribution factors py; 
25 Figure 2bT shows a graphical illustration of the decisions for updating the 

distribution quotas q i; and 

Figure 3t shows a formula for linear interpolation of qi. 
DETAILED DESCRIPTION OF THE PREF ERRED EMB ODIMENTS 
The method according to the present invention (normal load 
30 balancing=NLB) is a load balancing method in which quotas are set and which runs 



32 



on a multiprocessor system, in particular in a switching center of a commimication 
system, for distributing operating loads that arise between the respective other 
processorsraaad It is intended to ensure that lengthy unbalanced load situations are 
eliminated^ and as far as possible^ all requested tasks are processed in the shortest 
5 possible time. A particularly advantageous embodiment of this method will be 
described below. 

Each processor MPi where i=l,2,...,n carries a distribution quota qi, which 
fixes the proportion V of the distributable load which is actually to be distributed. 
Such a quota enables a softer entry or exit from the load distribution to other 
10 processors. Oscillation states and load fluctuations are avoided in this way. This 
maybe the case, for example, if a processor distributes so much load to another 
processor that the latter is in turn overloaded. 

The distribution quota qi is determined anew at each time interval CI. The 
only information required by the other processors MPk where k=l,...i-l,i+l,...n for 
15 each CI are load value indicators (balancing indicators) MPbij. These load value 
indicators are^ similarly to the load status values (load states)from the load control^ 
load states having the significances NORMAL, fflGH or OVERLOAD. While the 
load state is determined on the basis of the actually processed load Y; of the 
processor MPi, the load value indicator MPbii is determined from an estimation of 
20 the currently offered load Ai. The estimated offered load Ai may, due to load 
distribution, be considerably more than the actually processed load Yi and 
constitutes the crucial quantity which (in the form of the load value indicator 
MPbii) is made available as information by one processor MP; to the others MP^. 

In addition to the distribution quota qi, each MPi carries probabiHties pij 
25 which indicate the probability that, in the event of load distribution, load will be 
transferred from the i-th processor MPi to the j-th processor MPj. The probabilities 
are determined in such a way that if, for instance, the j-th processor MPj already has 
a large amount of load to process and, therefore, can only take up a small amount of 
additional load, the associated pij is less than the Pik for a free MPk. 
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Figure 1 illustrates the interaction of the py and qi. The double indexing "ij" 
of the characteristic quantities mean s that refers to the respective processor with the 
number of the first index (here i) in each case knows knowing a "column" of n 
values with the second index (here j). It should be noted that each processor only 
5 knows its relevant values (that is to say its column), overall a square matrix being 
known in the system. Thus, for example, pi, is the probability that load will be 
distributed from the i-th MP to the j-th MP if the i-th MP has too much load. 

In Figure 1, moreover, the actually processed load of the j-th processor MPj 
is designated by Yj, the estimated offered load is designated by Aj and that part of 
10 the load offer which can be shifted is designated by a. The load situation shown is 
overload (OVERLOAD) on MPi, there still being space for additional tasks on the 
MPk where k=2,3,4. The figure Figure 1 shows how the MPi processes a first part 
of the load itself and distributes the remainder a. Of this remainder a, the largest 
proportion goes to MPs and the smallest proportion to MP4, which, in this example, 
1 5 tfees already has a large amount of its own load to process. The loads which the 
MPk additionally receive besides that from MPi are not depicted. The width of the 
flow bars represents a measure of the magnitude of the load. 

The following algorithm is thus produced in accordance with the concept of 
the present invention: if the j-th processor MPj reports the balancing indicator 
20 NORMAL, the py is increased on the MPi respectively considered. The probability 
that this processor MP, will output load to MPj if it has to distribute load thus rises. 
If the balancing indicator HIGH is reported, then the py is decreased. If the 
balancing indicator OVERLOAD is reported, py is set to zero, with the result that 
no load is output to the j-th processor MPj. The distribution quota qi is changed 
25 following the determination of the py. If many of the py were able to be increased, 
then the sum of the Py over j is greater than 1 and there is evidently still space on 
the other processors MPk- The distribution quota qi can^ thus^ be changed according 
to the requirements of the processor (considered). 

The distribution quota qi is increased in the event of high load Yi on the processor 
30 MPi considered, and qi is decreased in the event of low load. If many of the pij have 
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been reduced, then the sum of the Py over j is less than 1 and the distribution quota 
qi must be reduced. 

An illustration of these decisions is represented in figures Figures 2 and 2b. 
The decision diagrams show the updating algorithms for pij (Figure 2a) and for 
5 distribution quota qi (Figure 2b), which are carried out in each time interval CI for 
the i-th processor MPj. 

In the load distribution method (NLB) according to the present invention, 
some parameters (constants) are required, the choice of which can greatly influence 
the behavior in specific load situations. In most cases this results in a conflict 
1 0 between a load distribution method which can react rapidly to load changes, and a 
stable load distribution method which does not tend toward oscillations and further 
distribution of tasks. In this case, "further distribution" means refers to the 
simultaneous distribution of dedicated load and the processing of extrinsic load on 
one processor. 

1 5 The following parameter changes bring about a more rapidly reacting NLB: 

- the relatively great alteration of qi where: 0.15<Cqi, 0.1<Cq2 

- the relatively great alteration of the py where: 0.25<pci, 0.25<pc2 

- the relatively late setting of the load indication values MPbii where: 
thresholdH>0.7 (i.e. report only in the event of relatively high load 'HIGH') to the 

20 other processors MPk. 

In detail, the preferred method thus proceeds as follows in a multiprocessor 
commimication computer: 

As the duration of the time interval (control interval) CI of the time frame 
with which the method iteratively proceeds, 1 to 2 seconds is preferably chosen in 
25 the case of the presently known multiprocessor systems appertaining to switching 
technology. It goes without saying that the time interval can be shortened with 
rising processor power. 

The quantities qi, pij, MPlSj and MPbij are updated in each control interval 

CI. 
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The actually processed load Y; of a processor MP, is determined as 
processor run time quantity, measured in erlangs. 

The estimated offered load Aj of a processor MPj is determined from the 
distribution quota q; of the current control interval CI and the estimated 
5 distributable proportion of an average task, for example the processing of a call. 
The following holds true: 

The number of processors MPi in the multiprocessor system is n. 
Ai := Yi/(l-qiV), where V is the distributable proportion of a call. 
MPlsj: load state of the i-th MP, can assume the values NORMAL, HIGH, 
10 OVERLOAD or EXTREME. The actually processed load Yi is used to calculate 
the load state. 

In order to avoid premature changes of the MPlsi, hystereses are introduced. 
If, for instance, the MPlsj is set from NORMAL to HIGH, it must be the case that 
Yi > thresholdN + A+, whereas, in order to get from HIGH to NORMAL, it must be 

1 5 the case that Yi < thresholdN - A.. This procedure is also known as the high water- 
low water method. In the case of EXTREME, the distribution method (load 
balancing) must be switched off for this processor MPi, for system engineering 
reasons relating to the switching center. 

thresholdN: is the normal load threshold - after taking a hysteresis into 

20 account, the MPls is recorded as NORMAL below the said threshold and as HIGH 
above said threshold. 

thresholdn: High load threshold - after taking a hysteresis and a load- 
dependent temporal delay (start indicator) into account, the MPls is recorded as 
HIGH below this threshold and as OVERLOAD above said threshold. 

25 The load indication value (balancing indicator) MPbii of the i-th processor 

MPi can assume the values NORMAL, HIGH or OVERLOAD. This value is 
calculated like the MPlsi, except that here, instead of the actual load Yi, the 
estimated offered load Ai is taken as a basis and other values are adopted for A+ and 
A., where A+ = A. = 0.02. 



36 



In addition, an Overload Level OLi of the processor MPi is determined, 
which can assume the values 0... 6 and is conceived as quantification of the 
overload state of the processor MPi. If the OLi>0, calls are rejected; the higher the 
value, the greater the prohability that a call will be rejected. 
5 The load which is to be distributed from MPi to MPj is expressed as a 

probabihty pij and can^ thus^ assume values between 0 and 1. 

The magnitude of the value pij is determined by the following criteria: 

- initialize pij where pij."(n-l)"^ 

- pii: 0, MPi should not distribute to itself. 

10 - If MPbij=NORMAL: pij -> pij + 0.25/n, j=l,.-,n, i^j. The old pij can be 

increased because there is still space on the processor MPj. 

- If MPbij==HIGH: pij pij - 0.25/n. The old pij must be decreased because 
MPj is utihzed to full capacity. 

- If MPbij-=OVERLOAD: pij = 0. No load should be output to overloaded 
15 processors MPn. 

The newly determined pij must still be normahzed: 
Set psum = sum (pij) over j=l,...,n and normahze (if psum > 0) where 
Pij Pij/P sum 

Afterward, the distribution quota qi is determined using the following 
20 criteria: 

- Initialization value: qi = 0.1 

- If the MPlsi ^EXTREME: qi = 0.1 . This MP is overloaded so severely that 
even its own proportion for a distributed call would overtax it. Therefore, no load 
balancing, rather only rejecting; load balancing is not practical, moreover, for 

25 system engineering reasons relating to the switching center. 

- If Psum > 1 , more load can evidently be distributed, qi can then be 
determined according to the requirements of the MPi, where: 

1. If the OLi > 0, increase qi in any case, where: qi min {qi + 0.15, 1} 

2. If Yi > thresholdH, increase qi, where: qi -> min {qi + 0.15, 1} 
30 3. If Yi < thresholdK, decrease qi, where: qi max {qi - 0.10, 0.1} 
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4. Otherwise, if thresholdN < Yi < thresholdn the following holds true: 

qi min{max{qi+(0.25/thresholdH-thresholdN)) * (Yi-thresholdN)-O.l, 0.1}, 

l.OK 

This is the linear interpolation between the above increase by 0.15 and the 
above decrease by 0.1. The formula is represented again more readably in figure 
Figure 3. 

- If psum< 1, evidently too much load was distributed and qi must be 
decreased, where: qi qj * psum- 

- The processor MPi distributes load to other processors MPk if it becomes 
the case that qi > 0.25. 

The method according to the present invention thus has the following 
properties and advantages: 

A very small information overhead exists between the processors 
participating in the load distribution method. Only a few, preferably three-value, 
load states are reciprocally known, which load states are updated and distributed 
only once per control interval. 

For each processor^ there is a quota which is updated in each control interval 
and regulates the proportion of the load which is to be distributed from the 
processor considered to the other processors involved. 

For each processor^ there are individual regulators which divide between the 
other processors the load that is to be distributed. 

The method is not only designed as a "fire- fighting measiu-e" which only 
takes effect when a processor reaches overload and, if appropriate, tasks (calls) are 
rejected, rather the load distribution commences earlier and in a softer fashion. As a 
result, continuous unbalanced load states can be processed better and with fewer 
rejected tasks (calls). 

In the method according to the present invention, the load states which are 
distributed to the other processors are consistently determined on the basis of the 
estimated offered load and not on the basis of the actually processed load. 
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The present method does not require a load balancing flag which regulates 
entry into the load distribution. The entry is regulated by way of the distribution 
quota qi. Furthermore, mutual dependencies between the load states and the load 
balancing flag have been eUminated as a result of the absence of a load balancing 
5 flag. As a result, the algorithm more easily can be subsequently adapted to 
changed conditions. 

The load-dependent alteration of the individual regulators (load distribution 
factors pij) takes place as a function of the number n of processors participating in 
the load distribution. Consequently, the method is independent of the number of 
10 processors involved. 

The load-dependent alteration of the distribution quotas and of the 
individual regulators per control interval takes place in such a way as to avoid 
excessively slow "creeping" to the optimum value. 

The load-dependent alteration of the individual regulators prevents the 
15 values from staying at the setting of the preceding load distribution period during a 
period without load distribution. Rather, there is regulation back to an initial 
setting. 

The inertia^ known from the prior art^ in the alteration of the quotas has 
been removed in order to enable easier tracking to the load situation that is actually 
20 present. 

Attention is supplementary drawn to the definition of a few terms in this 
application: 

The term or word element "quota" describes the fraction of a whole with a range of 
values between 0 and 1. 
25 The term ef or word element "state" describes the instantaneous situation or 

the instantaneous current value of a quantity. Thus, e.g.^ the load state of a 
processor is to be understood as the value of the current load of the processor. 

The term hysteresis defines the dependence of a current quantity on its 
earlier values, its history or its time characteristic. 
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Although the present invention has been described with reference to specific 
embodiments, those of skill in the art will recognize that changes may be made 
thereto without departing from the spirit and scope of the invention as set forth in 
the hereafter appended claims. 
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Abstract 

ABSTR ACT OF THE DTSCI.OSIJRE 

Load di s tribution method of a multiprocessor system, and multiprocessor system 

The invention relates to a A method for load distribution in a real-time 
multiprocessor system and to a multiprocessor system, each processor carrying a 
distribution quota which fixes the proportion of the distributable load which is 
actually to be distributed. The distribution quota is determined anew at time 
intervals. The only information required by the other processors for each time 
interval are load value indicators, which depend on an estimated load. Probabilities 
indicating how load is transferred from one processor to the others during load 
distribution are additionally carried. Afterward, on the basis of its distribution quota 
and its load distribution factors, each processor distributes its distributable load to 
other processors if it distribution quota exceeds a predetermined value. 

Figure 1 
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Load distribution method of a multiprocessor system, 
and multiprocessor system 

5 

The invention relates to a method for load 
distribution in a multiprocessor system, in particular 
in a multiprocessor system of a communication system, 
in which tasks that arise can be processed by a 

10 plurality of processors MPi (where 1=1, 2,,.., n) under 
real-time conditions, and to a multiprocessor system, 
in particular of a communication system, having a load 
distribution mechanism. 

A similar method for load distribution in a 

15 multiprocessor system, in particular in a 
multiprocessor system of a communication system, is 
disclosed for example in the applicant's European 
patent application EP 0 645 702 Al . This document 
discloses a method for load balancing in a 

20 multiprocessor system, in particular a multiprocessor 
system of a communication system, in which tasks that 
arise can be processed by a plurality of processors 
under real time conditions, in which case, in order to 
perform the load balancing, generally the following 

25 method steps are mentioned: 

- each processor determines its load state in 
the form of a quantified magnitude, 

- the load states of the other processors are 
communicated to each processor within a time frame, 

30 - depending on its load state exceeding a 

specific magnitude and depending on the load states of 
the remaining processors, each processor outputs at 
least a portion of the tasks arising in it to the 
remaining processors, and 
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- the output tasks are divided between the 
remaining processors in accordance with the load states 
thereof . 

In the exemplary embodiment;. the method is 
5 concretized to the effect that distribution quotas are 
calculated during operation continually and before 
entry into the load distribution, which in this case 
does not begin until after a specific overload has been 
reached, according to which distribution quotas the 

10 individual processors output their distributable load 
to other processors in the case of overload. If the 
system is permanently utilized to capacity in a 
nonuniform manner, then the load is distributed only in 
the event of overload of one or more processors. This 

15 is accompanied by unnecessary load rejection, however. 
Reducing the overload threshold to a lower value does 
not lead to a satisfactory result because then an 
unnecessarily large amount of load is distributed and 
oscillation states can arise. This situation emerges 

20 from the assumption made there that the overload or the 
nonuniform loading lasts for a short duration. 

Furthermore, reference is made to Evans D.J. et 
al., "Dynamic Load Balancing using Task Transfer 
Probabilities" in Parallel Computing, Vol. 19, No. 8, 

25 August 1, 1993, pages 897-916, which also presents a 
load distribution strategy taking account of load 
distribution probabilities, although exclusively 
currently measured values are taken into consideration. 
The temporal development of the parameters considered 

30 is not taken into account. 

It is an object of the invention, therefore, to 
specify an improved load distribution method for a 
multiprocessor system which commences in good time and 
in a "soft" fashion and thereby eliminates permanent 

35 unbalanced load states in the load offer without load 
rejection. Moreover, the intention is also to specify a 
corresponding multiprocessor system. 
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The object is achieved on the one hand by means 
of a method having the method steps of the first method 
claim and on the other hand by means of a 
multiprocessor system having the features of the first 
5 apparatus claim. 

Accordingly, the inventors propose a method for 
load distribution in a multiprocessor system, in 
particular in a multiprocessor system of a 
communication system, in which tasks that arise can be 
10 processed by a plurality of processors MPi (where 
i = l,2,...,n) under real-time conditions, having the 
following iterative method steps that are repeated at 
time intervals CI: 

- each processor MPi determines its actual load 
15 state Yi and estimates as a function of previously 

coimnunicated distribution quotas qi (old) (where 
qi = load proportion to be distributed, if possible, to 
other processors MPk) and the typically distributable 
proportion V of a typical task its offered load Ai, 
20 which leads to a multi-value load indication value 
(balancing indicator) MPbii, 

- each processor MPi indirectly or directly 
communicates its load indication value MPbii to the 
respective other processors MPk (where 

25 k = 1,2, . . .1-1,1+1, . . .n) , 

- each processor MPi determines its load 
distribution factors pij (where j = l,2,...n) as a 
function of the load indication values MPbijc of said 
other processors MPk, 

30 - each processor MPi determines its 

distribution quota qi (new) as a function of its actual 
load state Yi and the load distribution factors Pij, 

- on the basis of its quota qi and its load 
distribution factors Pij, each processor MPi distributes 

35 its distributable load to other processors MPjc if its 
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distribution quota qi{new) exceeds a predetermined 
value qv. 

In order to estimate the offered load Ai of a 
processor MPi, it is advantageous to use the formula 
5 Ai:=Yi/ (l-qiV) . Ai and Yi can be specified in units of 
erlangs, while the variables qi and V are dimensionless 
fraction indications in accordance with their meaning. 

It is also advantageous to subdivide the multi- 
value load indication value (balancing indicator) MPbii 
10 into three discrete values, preferably the following 
demarcation with threshold values holding true: NORMAL 
for MPbii if the processor capacity utilization is from 
0 to 70%, HIGH for MPbii if the processor capacity 
utilization is from 70% to 85%, and OVERLOAD for MPbii 
15 if the processor capacity utilization is above 85%. 

It is also advantageous if a hysteresis is 
introduced in the case of a load state alteration on 
account of threshold value overshooting or threshold 
value undershooting in the case of rising or falling 
20 processor capacity utilization. 

Moreover, it may be advantageous if the load 
indication value (balancing indicator) MPbii is subject 
to a temporal hysteresis with regard to changes and 
thus experiences a certain inertia. Values of 1 to 2 
25 time intervals CI can advantageously be assumed as 
hysteresis limit. 

Further advantageous assumptions in the 
performance of the method according to the invention 
are: the typical distributable proportion V of a 
30 typical task shall be the average 
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or maximum proportion, and an average or maximum 
processing time of a task shall be assumed as the 
typical processing time of a task. In this case, the 
respective average value or maximum value of a 
5 proportion and respectively of a task can also 
advantageously be continually determined during the 
operating time and, if appropriate, be incorporated as 
a moving value and be adopted in updated form into the 
load distribution method. It is favorable here if the 

10 time duration over which the moving values are 
determined is long relative to the control interval CI. 

It is also particularly advantageous if the 
following holds true for the predetermined value qv of 
the distribution quota qi starting from which the 

15 processor MPi distributes distributable load to other 
processors MPt: 0.05<qv<0.3, preferably 0.1<qv<0.25, 
preferably qv=0.2. 

Furthermore, the method according to the 
invention can be configured particularly advantageously 

20 if the following criteria are satisfied in the 
calculation of the distribution quota qi: 

- Pii: =0 

- if MPbij corresponds to an average load, 
preferably MPbij = NORMAL, the following holds 

25 true: 

Pi j (new) = pij (old) +pci/n, for j=l,...,n and i^j 

- if MPbij corresponds to a high load, 
preferably MPbij=HIGH, the following holds true: 

Pij (new) = Pij (old) -pc2/n, for j=l,...,n and i^^^j 
30 - if MPbij corresponds to an overload, 

preferably MPbij =OVERLOAD, the following holds true: 
Pij (new) = 0 

- in which case the Pij (j=i,...,n) is normalized to 
1 with the sum psu^ of the pij and 

35 - as initialization value at the beginning of 

the distribution processes, all Pij, excluding pii, are 
identical . 
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As advantageous numerical values, 0.1<pci<0.5, 
preferably 0. 2<Pci<0 . 3 and preferably Pci=0.25 may be 
assumed for the constant Pci. Equally, it is 
advantageous to set 0.1<pc2<0.5, preferably 0.2<pc2<0.3, 
5 preferably pc2=0.25 for the constant Pc2 • Moreover, the 
initialization value of the pij at the beginning of the 
distribution processes can be set to be equal to (n-1)"''". 

Furthermore, the method according to the 
invention can be configured particularly advantageously 
10 if each processor Mpi determines a multi-value load 
status (load state) MPlsi on the basis of its actual 
current load Yi, and the following criteria are 
satisfied in the calculation of the load indication 
values MPbii: 

15 - if MPlSi corresponds to the highest load, 

preferably MPlSi=EXTREME, the following holds true: 
qi(new)=Cqi, 

- if Psum ^ 1 holds true: 

- if the actual load state Yi is greater than a 
20 predetermined value thresholda, qi is increased where 

qi=min{qi+Cqi, 1} , 

- if the actual load state Yi is less than a 
predetermined value thresholdw, q± is decreased where 
qi=max{qi-Cq2, Cqs}, where 0<Cq3<qv, preferably Cq3=0.1, 

25 - otherwise (thresholdu < Yi < thresholdn) , qi 

obtains an intermediate value between the two 
alternatives mentioned above, preferably by linear 
interpolation 

- if Psum ^ 1 holds true: qi (new) =qi (old) * Psum- 
30 With regard to the multi-value load status 

(load state) MPlSi, the assumption of four discrete 
values is proposed as being particularly preferred, the 
following preferably being assumed: NORMAL for 
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MPlSi if the processor capacity utilization lies below 
70%, HIGH for MPlsi if the processor capacity 
utilization is from 70% to 85%, OVERLAND for MPlsi if 
the processor capacity utilization lies above 85%, and 
5 EXTREME for MPlsi if the load state OVERLAND 
permanently prevails. In this case, two, is maybe 
advantageous if the load status (load state) MPlsi is 
subject to a hysteresis with regard to changes. Values 
of 1 to 2 time intervals CI can advantageously be 
10 assumed as hysteresis limit. 

For optimal configuration of the method, the 
following ranges of numbers of numerical values are 
preferred for the constant Cqi: 
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0.05<Cqi<0.3, preferably 0-l<Cqi<0.2, preferably 
Cqi=0.15, Moreover, preferably 0 . 05<Cq2<0 . 2 , preferably 
Cq2=0.10 can be assumed for the constant Cq2. 

With regard to the constant threshold^, the 
5 following is regarded as a preferred range of values: 
0 . 6< thresholdn <0.8, preferably thresholdN=0. 7 . 

With regard to the constant thresholdn, the 
following is regarded as a preferred range of values: 
0.7< thresholds <0.95, preferably thresholdH=0 . 85 . 

10 Another configuration of the method according 

to the invention provides for an overload value OLi of 
the processors MPi to be additionally determined in 
each time interval CI, which value is a measure of the 
magnitude of the overload and serves as a benchmark for 

15 overload rejection,, where OLi=0, 1, . . .m, OLi representing 
a quantification for the overload of the processor, and 
the distribution quota qi to be increased in any case 
if OLi>0 where qi (new) : =min { qi ( old) +Cqi, 1 } . 
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According to the invention, it is also possible 
to adapt the load distribution method to changing 
boundary conditions by the above-specified constants 
(qvA Pel, Pc2r qa, qc2, thresholdN, thresholdw, Cqi, Cq2, 
5 Cq3) being at least partly adapted during operation. 

The invention additionally proposes a 
multiprocessor system, in particular of a communication 
system, having a plurality of processors MPi (where 
i=l,2,...,n) for executing tasks that arise under real- 
10 time conditions, in which case: 

- each processor MPi has means for determining 
its actual load state Yi, and for estimating as a 
function of previously communicated distribution quotas 
qi{old) (where qi=load proportion to be distributed, if 

15 possible, to other processors MPk) and the typically 
distributable proportion V of a typical task its 
offered load Ai, which leads to a multi-value load 
indication value (balancing indicator) MPbii, 

- each processor MPi has means for indirectly 
20 or directly communicating its load indication value 

MPbii to the respective other processors MPk (where 
k = 1,2, . . .1-1,1+1, . . .n) , 

- each processor MPi has means for determining 
its load distribution probabilities pij (where 

25 j = l,2,...n) as a function of the load indication 
values MPbik of said other processors MPk, 

- each processor MPi has means for determining 
its distribution quota qi(new) as a function of its 
actual load state Yi and 

30 - each processor MPi has means for 

distributing, on the basis of its quota qi and its load 
distribution factors Pij , 
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its distributable load to other processors MPk if its 
distribution quota qi(new) exceeds a predetermined 
value qv. 

According to the invention, the multiprocessor 
5 system proposed above can be configured such that one 
of the abovementioned methods is implemented in each 
case, the implementation being effected by- 
corresponding programming of the processors. 

It should also be pointed out that the index 
10 (old) relates in each case to the values of the 
preceding iteration step, and the index (new) relates 
to the now current iteration step. 

The particular advantage of the method 
according to the invention and of the corresponding 
15 multiprocessor system is that, in contrast to the prior 
art mentioned in the introduction, it ensures a "soft" 
entry into the load distribution and, as a result, is 
more adaptable and less susceptible to unbalanced load 
situations, and oscillation states are avoided better. 
20 Ultimately, this reduces the probability of the 
rejection of tasks, in particular switching tasks. 

Further configurations, additional features and 
advantages of the invention emerge from the following 
description of a preferred exemplary embodiment with 
25 reference to the drawings. 

It is understood that the features of the 
invention that have been mentioned above and will be 
explained below can be used not only in the combination 
respectively specified but also in other combinations 
30 or by themselves, without departing from the scope of 
the invention. 
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Specifically, in the figures: 
Figure 1: shows a flow diagram of the arising and 

distributed load offer 
Figure 2a: shows a graphical illustration of the 

5 decisions for updating the load 

distribution factors pij 
Figure 2b: shows a graphical illustration of the 

decisions for updating the distribution 
quotas qi 

ID Figure 3: shows a formula for linear interpolation 

of qi. 

The method according to the invention (normal 
load balancing=NLB) is a load balancing method in which 
quotas are set and which runs on a multiprocessor 

15 system, in particular in a switching center of a 
communication system, for distributing operating loads 
that arise between the respective other processors, and 
is intended to ensure that lengthy unbalanced load 
situations are eliminated and as far as possible all 

20 requested tasks are processed in the shortest possible 
time. A particularly advantageous embodiment of this 
method will be described below. 

Each processor MPi where i=l,2,...,n carries a 
distribution quota qi, which fixes the proportion V of 

25 the distributable load which is actually to be 
distributed. Such a quota enables a softer entry or 
exit from the load distribution to other processors. 
Oscillation states and load fluctuations are avoided in 
this way. This may be the case, for example, if a 

30 processor distributes so much load to another processor 
that the latter is in turn overloaded. 

The distribution quota qi is determined anew at 
each time interval CI. The only information required by 
the other processors MPk where k=l , . . . i-1 , i + 1 , . . . n for 

35 each CI are load value indicators (balancing 
indicators) MPbii. These load value indicators 
are - similarly to the load status values (load states) 
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from the load control - load states having the 
significances NORMAL, HIGH or OVERLOAD. While the load 
state is determined on the basis of the actually 
processed load Yi of the processor MPi, the load value 
5 indicator MPbii is determined from an estimation of the 
currently offered load Ai. The estimated offered load Aj_ 
may, due to load distribution, be considerably more 
than the actually processed load Y± and constitutes the 
crucial quantity which (in the form of the load value 

10 indicator MPbii) is made available as information by 
one processor MPi to the others MP^. 

In addition to the distribution quota qi, each 
MPi carries probabilities pij which indicate the 
probability that, in the event of load distribution, 

15 load will be transferred from the i-th processor MPi to 
the j-th processor MPj . The probabilities are 
determined in such a way that if, for instance, the j- 
th processor MPj already has a large amount of load to 
process and, therefore, can only take up a small amount 

20 of additional load, the associated pij is less than the 
Pik for a free MP^. 

Figure 1 illustrates the interaction of the pij 
and qi. The double indexing "ij" of the characteristic 
quantities means that the respective processor with the 

25 number of the first index (here i) in each case knows a 
"column" of n values with the second index (here j). It 
should be noted that each processor only knows its 
relevant values (that is to say its column) , overall a 
square matrix being known in the system. Thus, for 

30 example, Pij is the probability that load will be 
distributed from the i-th MP to the j-th MP if the i-th 
MP has too much load. 

In Figure 1, moreover, the actually processed 
load of the j-th processor MPj is designated by Yj, the 

35 estimated offered load is designated by Aj and that 
part of the load offer which can be shifted is 
designated by a. The load situation shown is overload 
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(OVERLOAD) on MPi, there still being space for 
additional tasks on the MPk where k:=2,3,4. The figure 
shows how the MPi 
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processes a first part of the load itself and 
distributes the remainder a. Of this remainder a, the 
largest proportion goes to MP3 and the smallest 
proportion to MP4, which, in this example, thus already 
5 has a large amount of its own load to process. The 
loads which the MPk additionally receive besides that 
from MPi are not depicted. The width of the flow bars 
represents a measure of the magnitude of the load. 

The following algorithm is thus produced in 

10 accordance with the concept of the invention: if the j- 
th processor MPj reports the balancing indicator 
NORMAL, the pij is increased on the MPi respectively 
considered. The probability that this processor MPi 
will output load to MPj if it has to distribute load 

15 thus rises. If the balancing indicator HIGH is 
reported, then the pij is decreased. If the balancing 
indicator OVERLOAD is reported, pij is set to zero, with 
the result that no load is output to the j-th processor 
MPj. The distribution quota qi is changed following the 

2 0 determination of the pij . If many of the pij were able 
to be increased, then the sum of the pij over j is 
greater than 1 and there is evidently still space on 
the other processors MPk. The distribution quota qi can 
thus be changed according to the requirements of the 

25 processor (considered) . 

The distribution quota qi is increased in the event of 
high load Yi on the processor MPi considered, and qi is 
decreased in the event of low load. If many of the pij 
have been reduced, then the sum of the Pij over j is 

30 less than 1 and the distribution quota qi must be 
reduced. 

An illustration of these decisions is 
represented in figures 2 and 2b. The decision diagrams 
show the updating algorithms for pij (Figure 2a) and for 
35 distribution quota qi (Figure 2b) , which are carried 
out in each time interval CI for the i-th processor 
MPi. 
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In the load distribution method (NLB) according 
to the invention, some parameters (constants) are 
required, the choice of which 
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can greatly influence the behavior in specific load 
situations. In most cases this results in a conflict 
between a load distribution method which can react 
rapidly to load changes, and a stable load distribution 
5 method which does not tend toward oscillations and 
further distribution of tasks. In this case, "further 
distribution" means the simultaneous distribution of 
dedicated load and the processing of extrinsic load on 
one processor. 

10 The following parameter changes bring about a 

more rapidly reacting NLB: 

- the relatively great alteration of qi where: 

0.15<Cql, 0.1<Cq2 

- the relatively great alteration of the pij 
15 where: 0.2 5<pci, 0.25<pc2 

- the relatively late setting of the load 
indication values MPbii where: thresholdH>0 . 7 (i.e. 
report only in the event of relatively high load 
^HIGH' ) to the other processors MPk 

2 0 In detail, the preferred method thus proceeds 

as follows in a multiprocessor communication computer: 

As the duration of the time interval (control 
interval) CI of the time frame with which the method 
iteratively proceeds, 1 to 2 seconds is preferably 
25 chosen in the case of the presently known 
multiprocessor systems appertaining to switching 
technology. It goes without saying that the time 
interval can be shortened with rising processor power. 

The quantities qi, pij, MPlSi and MPbij are 
30 updated in each control interval CI. 

The actually processed load Yi of a processor 
MPi is determined as processor run time quantity, 
measured in erlangs. 

The estimated offered load Aj of a processor MPi 
35 is determined from the distribution quota qi of the 
current control interval CI and the estimated 
distributable proportion of an 
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average task, for example the processing of a call. 
The following holds true: 

The number of processors MPi in the 
multiprocessor system is n. 
5 Ai := Yi/d-qiV), where V is the distributable 

proportion of a call. 

MPlsi: load state of the i-th MP, can assume 
the values NORMAL, HIGH, OVERLOAD or EXTREME. The 
actually processed load Yi is used to calculate the 

10 load state. 

In order to avoid premature changes of the 
MPlSi, hystereses are introduced. If, for instance, the 
MPlSi is set from NORMAL to HIGH, it must be the case 
that Yi > thresholdN + A+, whereas, in order to get from 

15 HIGH to NORMAL, it must be the case that 
Yi < thresholdN - A-. This procedure is also known as 
the high water-low water method. In the case of 
EXTREME, the distribution method (load balancing) must 
be switched off for this processor MPi, for system 

20 engineering reasons relating to the switching center. 

thresholdN". is the normal load 

threshold - after taking a hysteresis into account, the 
MPls is recorded as NORMAL below the said threshold and 
as HIGH above said threshold. 

25 thresholdn: High load threshold - after taking 

a hysteresis and a load-dependent temporal delay (start 
indicator) into account, the MPls is recorded as HIGH 
below this threshold and as OVERLOAD above said 
threshold. 

30 The load indication value (balancing indicator) 

MPbii of the i-th processor MPi can assume the values 
NORMAL, HIGH or OVERLOAD. This value is calculated like 
the MPlSi, except that here, instead of the actual load 
Yi, the estimated offered load Ai is taken as a basis 

35 and other values are adopted for A+ and A-, where 
A+ = A- = 0.02. 
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In addition, an Overload Level OLi of the 
processor MPi is determined, which can assume the 
values 0... 6 and is conceived as quantification of the 
overload state of the processor MPi. If the OLi>0, calls 
5 are rejected; the higher the value, the greater the 
probability that a call will be rejected. 

The load which is to be distributed from MPi to 
MPj is expressed as a probability pij and can thus 
assume values between 0 and 1. 
10 The magnitude of the value pij is determined by 

the following criteria: 

- initialize pij where pij : = (n-1 ) '■'^ 

- pii". 0, MPi should not distribute to itself. 

- If MPbij=NORMAL: Pij -> Pij + 0.25/n, 
15 j=l,...,n, ±1^3. The old pij can be increased because 

there is still space on the processor MPj. 

- If MPbij=HIGH: Pij -» pij - 0.25/n. The old pij 
must be decreased because MPj is utilized to full 
capacity. 

20 - If MPbij=OVERLOAD: pij = 0. No load should be 

output to overloaded processors MPn- 

The newly determined pij must still be 
normalized: 

Set Psuiti' = sum (Pij) over j = l,...,n and normalize 
25 (if Psura > 0) where pij -> Pij/psum 

Afterward, the distribution quota qi is 
determined using the following criteria: 

- Initialization value: qi = 0.1 

- If the MPlSi =EXTREME: qi = 0.1. This MP is 
30 overloaded so severely that even its own proportion for 

a distributed call would overtax it. Therefore,, no load 
balancing, rather only rejecting; load balancing is not 
practical, moreover, for system engineering reasons 
relating to the switching center. 
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- If psum > If more load can evidently be 
distributed, qi can then be determined according to the 
requirements of the MPi, where: 

1. If the OLi > 0, increase qi in any case, 
5 where: qi min {qi + 0.15, 1} 

2. If Yi > thresholda, increase qi, where: 
qi min {qi + 0.15, 1} 

3. If Yi < threshold^, decrease qi, where: 
qi max {qi - 0.10, 0.1} 

10 4. Otherwise, if thresholdu < Yi < thresholdH 

the following holds true: 

qi -> min{max{qi+ {0.25/thresholdH-thresholdN) ) * 
(Yi-thresholdN) -0.1, 0.1}, 1.0} 

This is the linear interpolation between the 
15 above increase by 0.15 and the above decrease by 0.1. 
The formula is represented again more readably in 
figure 3. 

- If Psum< If evidently too much load was 
distributed and qi must be decreased, where: 

2 0 qi ^ qi * Psum. 

- The processor MPi distributes load to other 
processors MPk if it becomes the case that qi > 0.25. 

The method according to the invention thus has 
the following properties and advantages: 

25 A very small information overhead between the 

processors participating in the load distribution 
method. Only a few, preferably three-value, load states 
are reciprocally known, which load states are updated 
and distributed only once per control interval. 

30 For each processor there is a quota which is 

updated in each control interval and regulates the 
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proportion of the load which is to be distributed from 
the processor considered to the other processors 
involved. 

For each processor there are individual 
5 regulators which divide between the other processors 
the load that is to be distributed. 

The method is not only designed as a "fire- 
fighting measure" which only takes effect when a 
processor reaches overload and, if appropriate, tasks 
10 (calls) are rejected, rather the load distribution 
commences earlier and in a softer fashion. As a result, 
continuous unbalanced load states can be processed 
better and with fewer rejected tasks (calls) . 

In the method according to the invention, the 
15 load states which are distributed to the other 
processors are consistently determined on the basis of 
the estimated offered load and not on the basis of the 
actually processed load. 

The method does not require a load balancing 
20 flag which regulates entry into the load distribution. 
The entry is regulated by way of the distribution quota 
qi. Furthermore, mutual dependencies between the load 
states and the load balancing flag have been eliminated 
as a result of the absence of a load balancing flag. As 
25 a result, the algorithm can more easily be subsequently 
adapted to changed conditions. 

The load-dependent alteration of the individual 
regulators (load distribution factors pij ) takes place 
as a function of the number n of processors 
30 participating in the load distribution. Consequently, 
the method is independent of the number of processors 
involved . 

The load-dependent alteration of the 
distribution quotas and of the individual regulators 
35 per control interval takes place in such a way as to 
avoid excessively slow "creeping" to the optimum value. 
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The load-dependent alteration of the individual 
regulators prevents the values from staying at the 
setting of the preceding load distribution period 
during a period without load distribution. Rather, 
5 there is regulation back to an initial setting. 

The inertia - known from the prior art - in the 
alteration of the quotas has been removed in order to 
enable easier tracking to the load situation that is 
actually present. 
10 Attention is supplementarily drawn to the 

definition of a few terms in this application: 
The term or word element "quota" describes the fraction 
of a whole with a range of values between 0 and 1. 

The term of word element "state" describes the 
15 instantaneous situation or the instantaneous current 
value of a quantity. Thus, e.g. the load state of a 
processor is to be understood as the value of the 
current load of the processor. 

The term hysteresis defines the dependence of a 
20 current quantity on its earlier values, its history or 
its time characteristic. 
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1. A method for load distribution in a 

multiprocessor system, in particular in a 
multiprocessor system of a communication system, in 
which tasks that arise can be processed by a plurality 
of processors MPi (where i = l,2,...,n) under real-time 
conditions, having the following iterative method steps 
that are repeated at time intervals CI: 

- each processor MPi determines its actual 
current load Yi and estimates as a function of 
previously communicated distribution quotas qi(old) and 
the typically distributable proportion V of a typical 
task its offered load Ai, which leads to a multi-value 
load indication value (balancing indicator) MPbii, the 
distribution quota qi representing the load proportion 
which can be distributed to other processors MPk, 

- each processor MPi indirectly or directly 
communicates its load indication value MPbii to the 
respective other processors MP^ (where 
k = 1,2, .. . i-1, i+1, . . .n) , 

- each processor MPi determines its load 
distribution probabilities pij (where j = l,2,...n) as a 
function of the load indication values MPbik of said 
other processors MPk, 

- each processor MPi determines its 
distribution quota qi (new) as a function of its actual 
current load and the load distribution factors pij 
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- on the basis of its quota qi and its load 
distribution factors pij , each processor MPi distributes 
its distributable load to other processors MPk if its 
distribution quota qi(new) exceeds a predetermined 
value qv 

2. The method as claimed in the preceding claim 1, 

characterized in that the estimated offered load Ai of 
a processor MPi is calculated according to the formula 
Ai:=Yi/ (l-qiV) . 

O 3. The method as claimed in one of the preceding 

claims, characterized in that the multi-value load 
Co indication value (balancing indicator) MPbii can assume 

^ three discrete values, preferably the values NORMAL, 

3 HIGH and OVERLOAD, where NORMAL corresponds to a 

O processor capacity utilization of from 0 to 75%, HIGH 

y, corresponds to a processor capacity utilization of from 

D 70% to 85% and OVERLOAD corresponds to a processor 

J!t capacity utilization of from 85% to 100%. 

Q 4. The method as claimed in the preceding claim 3, 

characterized in that the load indication value 
(balancing indicator) MPbii is subject to a hysteresis 
with regard to changes. 

6. The method as claimed in one of the preceding 
claims, characterized in that the average or maximum 
distributable proportion of a typical task CallP is 
regarded as the typical distributable proportion V. 

7. The method as claimed in the preceding claim 6, 
characterized in that the average or 
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maximum distributable proportion of a typical task is 
continually determined as moving average or moving 
maximum value over a predetermined time period t^. 

8. The method as claimed in the preceding claim 7, 
characterized in that the following holds true for the 
predetermined time period to: to » CI. 

9. The method as claimed in one of the preceding 
claims, characterized in that an average or maximum 
task is assumed as the typical task. 

10. The method as claimed in the preceding claim 6, 
characterized in that the average or maximum task is 
continually determined as moving average or moving 
maximum value over a predetermined time period to- 

11. The method as claimed in the preceding 
claim 10, characterized in that the following holds 
true for the predetermined time period to: to » CI. 

12. The method as claimed in one of the preceding 
claims, characterized in that the following holds true 
for the predetermined value qv of the distribution 
quota qi starting from which the processor MPi 
distributes distributable load to other processors MPt: 
0.05<qv<0.3, preferably 0.1<qv<0.25, preferably qv=0 . 2 . 
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13. The method as claimed in one of the preceding 

claims, characterized in that the calculation of the 
distribution quota qi satisfies the following criteria: 

- Pii: = 0 

- if MPbij corresponds to an average load, 
preferably MPbij = NORMAL, the following holds 
true : 

Pi j (new) = pij (old) +pci/n, for j=l, . . . , and i^^^j 

- if MPbij corresponds to a high load, 
O preferably MPbij=HIGH, the following holds true: 

m Pij (new) = pij (old) -pc2/n, for j=l,...,n and i^^j 

ffi - if MPbij corresponds to an overload, 

,JS preferably MPbij=OVERLOAD, the following holds true: 

J3 Pij (new) = 0 

- in which case the pij (j=i,...,n) are normalized 
1=^ to 1 with the sum Psum of the pij and 

y - as initialization value at the beginning of 

ifi the distribution processes, all Pij, excluding pii, are 

O identical. 

^" 14. The method as claimed in the preceding 

claim 13, characterized in that the following holds 
true for the constant pci: 

0.1<pci<0.5, preferably 0.2<pci<0,3, preferably 

Pci=0.25. 

15. The method as claimed in one of the preceding 
claims 13-14, characterized in that the following holds 
true for the constant Pc2: 0.1<Pc2<0.5, preferably 
0.2<pc2<0.3, preferably Pc2=0.25. 

16. The method as claimed in one of the preceding 
claims 13-15, characterized in that the initialization 
value of the pij at the beginning of the distribution 
processes is set to be equal to (n-1)"''". 
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17 . The method as claimed in one of the preceding 

claims 13-16, characterized in that each processor MPi 
determines a multi-value load status (load state) MPlsi 
on the basis of its actual current load Yi, and the 
calculation of the load indication values MPbii 
satisfies the following criteria: 

- if MPlSi corresponds to the highest load, the 
following holds true: 

qi (new) =Cqi, 

- if Psum ^ 1 holds true: 

- if the actual current load Yi is greater 
than a predetermined value thresholdH, qi is increased 
where qi=min{ qi+Cqi, 1 } , 

- if the actual current load Yi is less 
than a predetermined value thresholds, qi is decreased 
where qi=max{ qi-Cqs, Cqs), where 0<Cq3<qv, preferably 

Cq3-0.1, 

- otherwise (thresholdN ^ Yi < thresholdn) , 
qi obtains an intermediate value between the two 
alternatives mentioned above, preferably by linear 
interpolation 

- if Psum ^ 1 holds true: qi (new) =qi (old) * Psum- 

18. The method as claimed in the preceding 
claim 17, characterized in that the load status (load 
state) MPlSi is subject to a hysteresis with regard to 
changes . 

19. The method as claimed in one of the preceding 
claims 17-18, characterized in that the multi-value 
load status (load state) MPlsi can assume four discrete 
values, preferably NORMAL (=0 to 0.7), HIGH (=0.7 to 
0.85), OVERLOAD (=0.8 5 to 1) and EXTREME (if load 
status over a plurality of CI OVERLOAD) . 
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20. The method as claimed in one of the preceding 
claims 17-19, characterized in that the following holds 
true for the constant Cqi: 0 . 05<Cqi<0 . 3, preferably 
0.1<Cqi<0.2, preferably Cqi=0.15. 

21. The method as claimed in one of the preceding 
claims 17-20, characterized in that the following holds 
true for the constant Cq2: 0 . 05<Cq2<0 - 2 , preferably 

Cq2=0.10. 

22. The method as claimed in one of the preceding 
claims 17-21, characterized in that the following holds 
true for the constant thresholdw: 
0.6 < thresholdN < 0.8, preferably thresholdw =0.7. 

23. The method as claimed in one of the preceding 
claims 17-22, characterized in that the following holds 
true for the constant thresholdn: 
0.7 < thresholdn < 0.95, preferably thresholdn = 0.85. 

24. The method as claimed in one of the preceding 
claims, characterized in that an overload value OLi of 
the processors MPi is additionally determined for the 
purpose of quantifying the overload state of the 
processors, which overload value is a measure of the 
magnitude of the overload, where OLi=0,l,...m and the 
distribution quota .qi is increased in any case if the 
magnitude of OLi becomes greater than 0 and 
qi (new) :=min{qi (old) +Cqi, 1} is set. 

25. A multiprocessor system, in particular of a 
communication system, having a plurality of processors 
MPi (where i=l,2,...,n) for executing tasks that arise 
under real-time conditions, in which case: 
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a 



- each processor MPi has means for determining 
its actual current load Yi and for estimating as 
function of previously communicated distribution quotas 
qi(old) and the typically distributable proportion V of 
a typical task its offered load Ai, which leads to a 
multi-value load indication value (balancing indicator) 
MPbii, the distribution quota qi representing the load 
proportion which can be distributed to other processors 
MPk, 

- each processor MPi has means for indirectly 
or directly communicating its load indication value 
MPbii to the respective other processors MPk (where 
k = 1,2, . . .1-1,1+1, . - .n) , 

- each processor MPi has means for determining 
its load distribution probabilities Pij (where 
j = l,2,...n) as a function of the load indication 
values MPbik of said other processors MPk, 

- each processor MPi has means for determining 
its distribution quota qi(new) as a function of its 
actual current load Yi and 

- each processor MPi has means for 
distributing, on the basis of its quota qi and its load 
distribution factors pij, its distributable load to 
other processors MPk if its distribution quota qi(new) 
exceeds a predetermined value qv 

26. The multiprocessor system as claimed in 

claim 25, characterized in that one of the methods as 
claimed in one of claims 1-24 is implemented. 
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Dictation A 



10 



15 



page 6 of highlighted copy- 
Each processor MPi determines a multi-value 
load status (load state) MPlsi on the basis of its 
actual current load Yi, and 

Dictation B 
page 7 

where Oli represents a quantification for the overload 
of the processor. 

Page 18 



Attention is supplementary drawn to the 
definition of a few terms in this application: 
20 The term or word element ""^quota" describes the fraction 
of a hole with a range of values between 0 and 1. 

The term of word element, "states" describes 
the instantaneous situation or the instantaneous 
current value of a quantity. Thus, e.g. the load state 
25 of a processor is to be understood as the value of the 
current load of the processor. 

The term hysteresis defines the dependence of a 
current quantity on its earlier values, its history or 
its time characteristic. 
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DeclarationTmd Power of Attorney For Patent Application 
Erklarung Fur Patentanmeldungen Mit Vollmacht 

German Language Declaration 



Als nachstehend benannter Erfinder erklare ich hiermit 
an Eides Statt: 



dass mein Wohnsitz, meine Postanschrift, und meine 
Staatsangehorigkeit den im Nachstehenden nach 
meinem Namen aufgefiihrten Angaben entsprechen, 



dass ich, nach bestem Wissen der ursprungliche, erste 
und alleinige Erfinder (falls nachstehend nur ein Name 
angegeben ist) oder ein ursprunglicher, erster und 
Miterflnder (falls nachstehend mehrere Namen 
aufgefuhrt sind) des Gegenstandes bin, fur den dieser 
Antrag gestellt wird und fur den ein Patent beantragt 
wird fur die Erfindung mit dam Titei: 

Lastverteilunasverfahren eines 

MultiDrozessorsvstems und 

MultlDrozessorsvstem 

deren Beschreibung 

(zutreffendes ankreuzen) 
□ hier beigefugt ist. 
iE! am 17.01.2000 als 
PCT internationale Anmeldung 

PCT Anmeldungsnummer PCT/EPOO/00317 

eingereicht wurde und am 

abgeandert wurde (falls tatsachlich abgeandert). 



Ich bestatige hiermit, dass ich den Inhalt der obigen 
Patentanmeldung einschliesslich der Anspruche 
durchgesehen und verstanden habe, die eventuell 
durch einen Zusatzantrag wie oben erwahnt abgean- 
dert wurde. 



Ich erkenne meine Pflicht zur Offenbarung irgendwel- 
cher Informationen, die fur die Prufung der vorliegen- 
den Anmeldung in Einklang mit Absatz 37, Bundes- 
gesetzbuch. Paragraph 1.56(a) von Wichtigkeit sind, 
an. 



Ich beanspruche hiermit auslandische Prioritatsvorteile 
gemass Abschnitt 35 der Zivilprozessordnung der 
Vereinigten Staaten, Paragraph 119 aller unten ange- 
gebenen Auslandsanmeldungen fur ein Patent oder 
eine Erfindersurkunde, und habe auch alle Auslands- 
anmeldungen fiir ein Patent oder eine Erfindersurkun- 
de nachstehend gekennzeichnet, die ein Anmelde- 
datum haben, das vor dem Anmeldedatum der 
Anmeldung liegt, fur die Prioritat beansprucht wird. 



As a below named inventor, I hereby declare that: 



My residence, post office address and citizenship are 
as stated below next to my name, 



I believe 1 am the original, first and sole inventor (if only 
one name is listed below) or an original, first and joint 
inventor (if plural names are listed below) of the 
subject matter which is claimed and for which a patent 
is sought on the invention entitled 



Load distribution method for a 

multiprocessor system and 

corresponding multiprocessor system 

the specification of which 

(check one) 

□ is attached hereto. 

M was filed on 17.01 .2000 as 

PCT international application 

PCT Application No. PCT/EP OO/00317 

and was amended on 

(if applicable) 

1 hereby state that I have reviewed and understand the 
contents of the above identified specification, including 
the claims as amended by any amendment referred to 
above. 



I acknowledge the duty to disclose information which is 
material to the examination of this application in 
accordance with Title 37, Code of Federal Regulations, 
§1 .56(a). 



1 hereby claim foreign priority benefits under Title 35, 
United States Code, §119 of any foreign application(s) 
for patent or inventor's certificate listed below and have 
also identified below any foreign application for patent 
or inventor's certificate having a filing date before that 
of the application on which priority is claimed: 



Form PTO-FB-240 (8-83) 
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Patent and Trademark Office-U.S. DEPARTMENT OF COMMERCE 



Prior foreign appplications 
Prioritat beanspruclit 



99101122.2 EP 

(Number) (Country) 
(Nummer) (Land) 



(Number) 
(Nummer) 



(Number) 
(Nummer) 



(Country) 
(Land) 



(Country) 
(Land) 



German Language Declaration 



21.01.1999 

(Day Month Year Filed) 
(Tag Monat Jahr eingereicht) 



(Day Month Year Filed) 
(Tag Monat Jahr eingereicht) 



(Day Month Year Filed) 
(Tag Monat Jahr eingereicht) 



Priority Claimed 



Yes 
Ja 



□ 

Yes 
Ja 



□ 
Yes 
Ja 



□ 
No 
Nein 



□ 
No 
Nein 



□ 
No 
Nein 



Ich beanspruche hiermit gemass Absatz 35 der Zivil- 
prozessordnung der Vereinigten Staaten, Paragraph 
120, den Vorzug aller unten aufgefuhrten Anmel- 
dungen und falls der Gegenstand aus jedem Anspmch 
dieser Anmeidung nicht in einer fruheren 
amerikanischen Patentanmeldung laut dem ersten 
Paragraphen des Absatzes 35 der Zivilproze3ordnung 
der Vereinigten Staaten, Paragraph 122 offenbart ist, 
erkenne ich gemass Absatz 37, Bundesgesetzbuch, 
Paragraph 1.56(a) meine Pflicht zur Offenbarung von 
informationen an, die zwischen dem Anmeldedatum 
der fruheren Anmeidung und dem nationalen Oder PCT 
internationalen Anmeldedatum dieser Anmeidung 
bekannt geworden sind. 



I hereby claim the benefit under Title 35. United States 
Code. §120 of any United States application(s) listed 
below and, insofar as the subject matter of each of the 
claims of this application is not disclosed in the prior 
United States application in the manner provided by 
the first paragraph of Title 35, United States Code, 
§122, I acknowledge the duty to disclose material 
information as defined in Title 37, Code of Federal 
Regulations, §1 .56(a) which occured between the filing 
date of the prior application and the national or PCT 
international filing date of this application. 



PCT/EPOO/00317 
(Application Serial No.) 
(Anmeldeseriennummer) 



(Application Serial No.) 
(Anmeldeseriennummer) 



17.01.2000 

(Filing Date D, M, Y) 
(Anmeldedatum T, M, J) 



(Filing Date D,M,Y) 
(Anmeldedatum T, M; J) 



anhangig 

(Status) 
(patentiert, 



Ich erklare hiermit, dass aile von mir in der vorliegen- 
den Erkiarung gemachten Angaben nach meinem 
besten Wissen und Gewissen der vollen Wahrheit 
entsprechen, und dass ich diese eidesstattliche Erkia- 
rung in Kenntnis dessen abgebe, dass wissentlich und 
vorsatzlich falsche Angaben gemass Paragraph 1001, 
Absatz 18 der Zivilprozessordnung der Vereinigten 
Staaten von Amerika mit Geldstrafe belegt und/oder 
Gefangnis bestraft werden koennen, und dass derartig 
wissentlich und vorsatzlich falsche Angaben die Gui- 
tigkeit der vorliegenden Patentanmeldung oder eines 
darauf erteilten Patentes gefahrden konnen. 



pending 

(Status) 

(patented, pending, 
abandoned) 



(Status) 

(patented, pending, 
abandoned) 



I hereby declare that all statements made herein of my 
own knowledge are true and that all statements made 
on information and belief are believed to be true, and 
further that these statements were made with the 
knowledge that willful false statements and the like so 
made are punishable by fine or imprisonment, or both, 
under Section 1001 of Title 18 of the United States 
Code and that such willful false statements may 
jeopardize the validity of the application or any patent 
issued thereon. 
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Patent and Trademark Office-U.S. DEPARTMENT OF COMMERCE 



VERTRETUNGSVOLLMACHT: Als benannter Erfinder 
beauftrage ich hiermit den nachstehend benannten 
Patentanwalt (oder die nachstehend benannten 
Patentanwalte) und/oder Patent-Agenten mit der 
Verfolgung der vorliegenden Patentanmeldung sowie 
mit der Abwicklung alier damit verbundenen Geschafte 
vor dem Patent- und Warenzeichenamt: (Name und 
Registrationsnummer anfuhren) 
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POWER OF ATTORNEY: As a named inventor, I 
hereby appoint the following attomey(s) and/or 
agent(s) to prosecute this application and transact all 
business in the Patent and Trademark Office 
connected therewith, (list name and registration 
number) 



And I hereby appoint 



Customer No. 



Telefongesprache bitte richten an: 
(Name und Telefonnummer) 



Direct Telephone Calls to: (name and telephone 
number) 

Ext. 



Postanschrift: 



Send Correspondence to: 



Bell, Boyd & Lloyd LLC 

Three First National Plaza , 70 West Madison Street , Suite 3300 60602-4207 Chicago, Illinois 
Telephone: (001) 312 372 11 21 and Facsimile (001) 312 372 20 98 

or 

Customer No. 



Voller h4ame des einzigen Oder ursprOnglichen Erfinders: 

PETER HANSELKA i - 


Full name of sole or first inventor: 

PETER HANSELKA 


Unter^hrift des Erfinders . / Datum 


Inventor's signature n Date 


Wolinsitz 

MUENCHEN, DEUTSCHLAND 


Residence 

MUENCHEN. GERMANY 


Staatsangeh6rigl<eit 
DE 


Citizenship 

DE 


Postanschrift 

PIDINGERSTR. 13 


Post Office Addess 

PIDINGERSTR. 13 


81379 MUENCHEN 


81379 MUENCHEN 


Voller Name des zweiten Miterfinders (falls zutreffend): 
GABRIELE RAICHLE ^ - «o 


Full name of second joint inventor, if any: 

GABRIELE RAICHLE„ 


Unterschrift d^ Erfinders Datum 


SecMd Inventor's signature Date 


Wohnsifi? 

MUENCHEN, DEUTSCHLAND 


Residence 

MUENCHEN, GERMANY aoe.^ 


Staatsangehorlgkeit 

DE 


Citizenship 

DE 


Postanschrift 

AMBACHERSTR. 39 


Post Office Address 

AMBACHERSTR. 39 


81476 MUENCHEN 


81476 MUENCHEN 



(e/tte entsprechende Informationen und Unterschriften im (Supply similar information and signature for third and 
Falle von dritten und weiteren Miterfindern angeben). subsequent joint Inventors). 
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Voller Name des dritten Miterfinders: 

Dr. CHRISTOPHER ROSS :s - 


Full name of third joint inventor: 

Dr. CHRISTOPHER BOSS 


Untersohrift des Erfinderar) Datum 


Inventof s signature^ Date 

(^d)^ rum n'^y.^ooi, 


Wohnsitz 

MUENCHEN, DEUTSCHLAND 


Residence " 

MUENCHEN, GERMANY X^^^ 


Staatsangehdrigkeit 

DE 


Citizenship 

DE 


Postansch rift 

KIRCHENSTR. 89 


Post Office Address 

KIRCHENSTR. 89 


81675 MUENCHEN 


81675 MUENCHEN 


Voller Name des vierten Miterfinders: 


Full name of fourth joint inventor: 


Untersohrift des Erfinders Datum 


Inventor's signature Date 


Wohnsitz 


Residence 


Staatsangeh6rigl<eit 


Citizenship 


Postanschrift 


Post Office Address 






Voller Name des fOnften Miterfinders: 


Full name of fifth joint inventor: 


Untersohrift des Erfinders Datum 


Inventor's signature Date 


Wohnsitz 


Residence 


Staatsangeh6rigl<eit 


Citizenship 


Postanschrift 


Post Office Address 






Voller Name des sechsten Miterfinders: 


Full name of sixth joint inventor: 


Untersohrift des Erfinders Datum 


Inventor's signature Date 


Wohnsitz 


Residence 


SfaatsangehSrigkeit 


Citizenship 


Postanschrift 


Post Office Address 







[Bitie entsprechende Informationen und Unterschriften im (Supply similar information and signature for third and 



Falle von dritten und weiteren Miterfindern angeben). subsequent joint inventors). 
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